Handle system time move backward case in timer task processing

[yycptt]

What changed?

• Handle system time move backward case in timer task processing by using max(now(), task visibility timestamp)

Why?

• Monotonic time is used in time.Time comparison only when both operands have monotonic time value. In our case, the timestamp stored/derived in/from mutable state doesn't have monotonic, thus the comparison logic will use wall clock time to decide if a timer task should be processed. If wall clock move backwards (after we verifies that now() > task visibility timestamp when submitting the task to the task scheduler), the timer task will be dropped and cause workflow to stuck.

How did you test it?

• Unit tests

Potential risks

• In worst case, we will execute a timer task logic earlier than expected.

Documentation

Is hotfix candidate?

• Could be. But the bug should be very rare as the use millisecond precision when doing time comparison in timer task processing.

[bergundy]

What's the difference between visibility timestamp and fire time and why take that here?

[yycptt]

They are the same for Scheduled tasks. For immediate tasks, fire time is always 0. Since this function is only meant to be invoked by scheduled tasks, I think FireTime is better as it won't have any impact on immediate tasks.

[bergundy]

Shouldn't we use referenceTime to allow progressing beyond the visibility timestamp in case task processing is delayed for a while?

[yycptt]

The goal here I believe is to clear the heartbeat timer task status only when we have to. Using referenceTime means we will clear the flag more often and causing more heartbeat timer task being created.