Add insight about span recording and propagation

[codefromthecrypt]

here are some notes I thought about when doing the async span investigation. openzipkin/zipkin#1189 (comment)

While rough, I expect these notes to be better than nothing, and can help us produce better documentation, particularly the instrumentation section. Feel free to break this apart.

Span recording is when timing information or metadata is structured and reported out to zipkin.

• zipkin merges spans together that share the same traceid and span id. The most common case of this is to merge a span reported by both the client (cs, cr) and the server (sr, ss)
• Zipkin adds Span.timestamp and duration at query time, if no authoritative values were stored.
  • instrumentation should implicitly set span.timestamp and duration when bookend annotations are present. For example, a client RPC span should have its span timestamp and duration set according to "cs" and "cr"
  • instrumentation should not implicitly set span.timestamp and duration when incomplete or otherwise unknown annotations are present. For example, if a client only logs "cs" and not "cr", you could set Span.timestamp, but not duration as there's not enough information.

Span propagation is when trace identifiers are sent in-band across to the next hop. The next hop uses the trace, parent, span id, and sampled value it receives. In other words, the sender is in control. If the sender wants the next hop to log into the same span, it propagates the current span id. If the sender wants the next hop to log into a child span, it propagates a child span id.

This is important when spans that are not client-server in nature. Taking the example of Kafka, the sender isn't sending to the consumer, they are sending to Kafka. This send activity is better modeled as a separate span, so if propagating a span id, it would make sense to propagate a child span. When the consumer picks up, which might be minutes later, if at all, it resumes the child span. Modeling producer/consumer as two separate spans highlights the non-blocking activity that's taking place.