Can't seem to make opentelemetry work

[RemiKalbe]

Hey,

I can't seem to make tonic work with distributed tracing. I have a client and a service, and the trace of the client is not propagated to the server.

.trace_fn(|_| {
            let span = tracing::info_span!("coordinator_registry_server", trace_id = field::Empty);

            // opentelemetry::Context -> opentelemetry::trace::Span
            use opentelemetry::trace::TraceContextExt as _;
            // tracing::Span -> opentelemetry::Context
            use tracing_opentelemetry::OpenTelemetrySpanExt as _;

            let trace_id = tracing::Span::current()
                .context()
                .span()
                .span_context()
                .trace_id()
                .to_string();

            span.record("trace_id", &field::display(&trace_id));

            span
        })

This end up printing the following: "trace_id":"00000000000000000000000000000000". Note that the code retrieving the trace_id works on the client, I get it's trace_id, it's on the server that I get this. I have no clue where the issue lies.

Obviously there is also nothing on grafana

And this is how I setup tracing in main.rs

global::set_text_map_propagator(TraceContextPropagator::new());

    let config = config::Config::default();

    let env_filter = EnvFilter::try_from_default_env()
        .or_else(|_| EnvFilter::try_new("info"))
        .unwrap();

    let exporter = opentelemetry_otlp::new_exporter()
        .tonic()
        .with_endpoint(config.env.opentelemetry_endpoint.clone());

    let trace_config =
        opentelemetry::sdk::trace::config().with_resource(opentelemetry::sdk::Resource::new(vec![
            opentelemetry::KeyValue::new("service.name", "coordinator-registry"),
            opentelemetry::KeyValue::new("service.namespace", "cloud-conference"),
        ]));

    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(exporter)
        .with_trace_config(trace_config)
        .install_batch(opentelemetry::runtime::Tokio)
        .expect("Failed to create opentelemtry pipeline");

    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

    let formatting_layer =
        BunyanFormattingLayer::new("coordinator-registry".to_string(), std::io::stdout);

    let subscriber = Registry::default()
        .with(env_filter)
        .with(telemetry)
        .with(JsonStorageLayer)
        .with(formatting_layer);

    tracing::subscriber::set_global_default(subscriber)
        .expect("Failed to install `tracing` subscriber.");

[LucioFranco]

Im not totally sure since I have yet to use the otel and tracing stuff. Maybe this question is better to ask in those repos since I don't think this is related to tonic except for the trace_fn part?

[RemiKalbe]

Tldr: opentelemetry context has to be injected into the request by the client, and then on the server it needs to be extracted, opentelemetry doesn't do this by itself, it's up to the crate, here Tonic to implement it.

[LucioFranco]

I believe the integration point is not tonic but its tower? Which looks like someone has already worked on it https://github.com/slickbench/tower-opentelemetry/blob/main/src/lib.rs. There is maybe opportunity for someone to write a otel crate that takes grpc headers and does stuff with that. That said, tonic should not have specific metric/tracing impls beyond having tracing which I am ok with adding new tracing points which tonic currently lacks.

[RemiKalbe]

It looks like this crate is sadly unmaintained. After browsing a bit I've found this discussion tokio-rs/axum#750 and it looks like otel on Tower isn't coming anytime soon. I think building a separate crate for tonic is the way to go, but I have no knowledge of Tonic, is it even possible to build a separate crate that allows otel tracing?

[LucioFranco]

Tonic works on top of tower so what ever would work for tower would work for tonic that is my suggestion.

[flokli]

I ran into the same problem, and first only found no pointers from #1184 (now added a comment linking to here).

How would I be able to use tower to make sure all my grpc clients propagate their tracing spans into their Request metadata?