is jaeger_collector_spans_received_total metric accurate?

[yasharne]

Hi
We have some microservices in our infrastructure and use jaeger for tracing of these services (and they have propagation enabled)
When we check the rate of jaeger_collector_traces_received_total in two minutes, it shows 16 traces for this service, but when we check the rate of jaeger_collector_spans_received_total metric in two minutes, it says this service have around 16k spans for this service.

When we check the jaeger UI, the spans seems normal (the filtered service is the one in the violet) :

We have deployed jaeger in microservice mode with Cassandra as backend.

[dosubot[bot]]

Hey @yasharne! I'm here to help you with your issue. Let's figure this out together.

Yes, the jaeger_collector_spans_received_total metric is accurate. According to the documentation, the /metrics endpoint can be used to check if spans for specific services were received, and the jaeger_collector_spans_received metric is specifically mentioned as one of the metrics of special interest. The documentation states that this metric should have similar values to jaeger_collector_spans_saved_by_svc for the same service, indicating its reliability.

Additionally, the documentation provides troubleshooting steps to ensure that spans are being correctly processed and stored, such as verifying the sampling strategy, using the logging reporter, bypassing the Jaeger Agent, and increasing the logging in the backend components. These steps can help diagnose and resolve any issues that might affect the accuracy of the jaeger_collector_spans_received_total metric.

The following metrics are of special interest:

    jaeger_collector_spans_received
    jaeger_collector_spans_saved_by_svc
    jaeger_collector_traces_received
    jaeger_collector_traces_saved_by_svc

The first two metrics should have similar values for the same service. Similarly, the two `traces` metrics should also have similar values. For instance, this is an example of a setup that is working as expected:

    jaeger_collector_spans_received{debug="false",format="jaeger",svc="order"} 8
    jaeger_collector_spans_saved_by_svc{debug="false",result="ok",svc="order"} 8
    jaeger_collector_traces_received{debug="false",format="jaeger",svc="order"} 1
    jaeger_collector_traces_saved_by_svc{debug="false",result="ok",svc="order"} 1

For further troubleshooting, you can refer to the detailed steps in the troubleshooting guide, which includes verifying the sampling strategy, using the logging reporter, bypassing the Jaeger Agent, and increasing the logging in the backend components ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[yasharne]

@dosu this metric doesn't exists jaeger_collector_spans_saved_by_svc

[dosubot[bot]]

The jaeger_collector_spans_saved_by_svc metric might not exist in your Jaeger setup because the spans are not being processed and stored correctly. Here are some steps to troubleshoot this issue:

1. Verify the Sampling Strategy: Ensure that the sampling strategy is configured correctly. For development or low-traffic scenarios, set the JAEGER_SAMPLER_TYPE environment variable to const and the JAEGER_SAMPLER_PARAM to 1 to sample every trace.

2. Use the Logging Reporter: Enable logging of spans by setting the JAEGER_REPORTER_LOG_SPANS environment variable to true. This will help you verify if spans are being reported by the client.

3. Bypass the Jaeger Agent: Configure the client to send spans directly to the Collector by setting the JAEGER_ENDPOINT environment variable to the Collector's endpoint (e.g., http://jaeger-collector:14268/api/traces).

4. Increase Logging in Backend Components: Start the Jaeger Agent and Collector with the --log-level=debug flag to get detailed logs about the spans being received and processed.

5. Check the /metrics Endpoint: Use the /metrics endpoint to verify if spans for specific services are being received. For example:

   curl http://jaeger-collector:14269/metrics

   Look for the following metrics:

   jaeger_collector_spans_received
   jaeger_collector_traces_received
   jaeger_collector_traces_saved_by_svc
   

If the jaeger_collector_spans_saved_by_svc metric is missing, it indicates that spans are not being saved by the service, which could be due to issues in the sampling strategy, reporting configuration, or network setup ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[yurishkuro]

The "traces" metric is computed from root spans only (spans with null parent). It is expected that it would be different (smaller) from "spans" metric, but 16 vs 16k doesn't sound quite right since your traces do not seem to be that large (screenshot shows about 30 spans per trace). Another possibility is if you have issues with collection where root spans are not making it to the collector and not included in the "traces" metric.