[#1962] Add newCallbackExecutor to interceptors and override for OpenTracing

[ghaskins]

This change adds newCallbackExecutor to the OutboundInterceptor interface, overriding the OpenTracing behavior. This allows the OpenTracing interceptor to transfer any active trace/span from the point of creation to the point of deferred execution, thus preserving the span within the callback.

Without this, spans become disjoint across Promise.thenCompose, due to the nature of the deferring the execution of the .thenCompose to a callback thread.

Fixes #1962

Checklist

1. Closes Tracing context does not propagate into .thenCompose #1962

2. How was this tested:
   A unit test was developed first to confirm the issue. The solution was iterated over until the unit test passed satisfactorily. The unit-test is included in the MR for inclusion with the code-base

3. Any docs updates needed?
   No.

[cretz]

Hrmm, this is a very specific interceptor-oriented solution to a general problem of contextual state across promises. I wonder if a WorkflowAsyncLocal is best which behaves like thread local, but also propagates across promises. This is the common solution to this problem in other languages, make a way to propagate state across async (Go is context, Python is contextvars, .NET is AsyncLocal, etc).

[ghaskins]

@cretz, if I understand what you are saying, I think we'd still need to use interception at some level unless OTEL can be made to utilize the WorkflowAsyncLocal instead of ThreadLocal because something would need to initialize the state in the callback context, which currently runs outside of the RootThread complex. I'm not saying OTEL can't use WorkflowAsyncLocal; I just don't know

[mfateev]

I believe this change is not backward compatible and can break determinism.

[ghaskins]

@mfateev are you referring to the forEach or the executor deferment?

[ghaskins]

I should add: Executor makes no assumptions about async/sync execution models...I chose it because it seemed to have the right abstraction where the context creation is decoupled from the execution of the Runnable. However, I fully expect any interceptor only to use synchronous execution to avoid changing the model. I could make this more explicit or use a different abstraction if we think the use of Executor confusingly suggests async or promotes abuse.

[ghaskins]

I added documentation to clarify the synchronous requirement and changed to a plain for() loop.

[mfateev]

My point is that switching from direct call to an executor that might change the order of execution of these calls can lead to non determinism. Such changes should be protected by versioning/patching.

[ghaskins]

Ah, I see. FWIW: I was careful to design the executor chain in a way that should respect the original order of interceptors, and I retained the original order of the for loop as well. I suspect your concern would be if someone added an interceptor that creates an NDE situation, but I think that if that were the case, they should follow patching procedures in the intercept code. I don't think we could solve that at this level. Thoughts?

[ghaskins]

@mfateev I looked, and I don't see any apparent examples of SDK code using patching internally. Since I don't believe the PR code changes the ordering, the only NDE potential that I see would be if an interceptor introduced a determinism issue in its handler. However, that would also concern any existing interception points, so it's unclear if this proposed one should be treated differently.

This is a blocking issue for me, so I am highly motivated to see it get merged. So please let me know what you'd like to see done here, and I will do my best to make it happen.