How to send application trace data from OpenTelemetry to Jaeger in Kubernetes?

[seikyo-cho-lvgs]

I installed opentelemery collector by helm

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=deployment --set image.repository="otel/opentelemetry-collector-k8s" --set command.name="otelcol-k8s" -f values.yaml

values.yaml

mode: deployment

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  exporters:
    otlp:
      endpoint: jaeger-collector.default.svc.cluster.local:14250
      tls:
        insecure: true
    debug:
      verbosity: detailed
  service:
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [otlp]
      metrics:
        receivers: [otlp]
        exporters: [debug]
      logs:
        receivers: [otlp]
        exporters: [debug]

Then installed Jaeger by helm

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger

Port forward for local application accessing

kubectl port-forward svc/opentelemetry-collector 4317:4317
kubectl port-forward svc/opentelemetry-collector 4318:4318

Run an application to point to the port

/*app.ts*/
import express, { Express } from 'express';

const PORT: number = parseInt(process.env.PORT || '8080');
const app: Express = express();

function getRandomNumber(min: number, max: number) {
  return Math.floor(Math.random() * (max - min) + min);
}

app.get('/rolldice', (req, res) => {
  res.send(getRandomNumber(1, 6).toString());
});

app.listen(PORT, () => {
  console.log(`Listening for requests on http://localhost:${PORT}`);
});

/*instrumentation.ts*/
import * as opentelemetry from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'test-service',
});

const sdk = new opentelemetry.NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4317',
    // optional - collection of custom headers to be sent with each request, empty by default
    headers: {},
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://localhost:4317',
      headers: {}, // an optional object containing custom headers to be sent with each request
    }),
  }),
  resource: resource,
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

// Run
npx ts-node --require ./instrumentation.ts app.ts

After access the app several times, the opentelemetry-collector pod got error log:

2024-07-19T08:50:13.705Z	error	exporterhelper/queue_sender.go:90	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:90
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43

The log in jaeger-collector pod:

{"level":"info","ts":1721717166.6132424,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams received bogus greeting from client: \\\"GET / HTTP/1.1\\\\r\\\\nHost: lo\\\"\"","system":"grpc","grpc_log":true}

Why can't send grpc data from OpenTelemetry Collector to Jaeger service?

[dosubot[bot]]

@seikyo-cho-lvgs こんにちわ! I'm here to assist you with any questions or issues you have. Feel free to ask for help!

To send application trace data from OpenTelemetry to Jaeger in Kubernetes and resolve the error "rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", you need to ensure that the Jaeger collector is configured to enable the OpenTelemetry Protocol (OTLP) receiver.

Here is how you can configure the Jaeger collector to enable the OTLP receiver:

1. Update the Jaeger Helm Chart Configuration:
   Ensure that the Jaeger collector is configured to enable the OTLP receiver. You can do this by setting the collector.otlp.enabled option to true in the Jaeger Helm chart values.

   Example configuration snippet for the Jaeger Helm chart (values.yaml):

   collector:
     otlp:
       enabled: true
       grpc:
         host-port: ":4317"  # Default port for OTLP gRPC

2. Apply the Configuration:
   Update your Jaeger installation with the new configuration:

   helm upgrade jaeger jaegertracing/jaeger -f values.yaml

3. Verify the OpenTelemetry Collector Configuration:
   Ensure that your OpenTelemetry Collector is configured to send data to the Jaeger collector's OTLP gRPC endpoint. Your values.yaml for the OpenTelemetry Collector should look like this:

   mode: deployment
   
   config:
     receivers:
       otlp:
         protocols:
           grpc:
             endpoint: 0.0.0.0:4317
           http:
             endpoint: 0.0.0.0:4318
     exporters:
       otlp:
         endpoint: jaeger-collector.default.svc.cluster.local:4317
         tls:
           insecure: true
       debug:
         verbosity: detailed
     service:
       pipelines:
         traces:
           receivers: [otlp]
           exporters: [otlp]
         metrics:
           receivers: [otlp]
           exporters: [debug]
         logs:
           receivers: [otlp]
           exporters: [debug]

4. Restart the Services:
   After updating the configurations, restart the OpenTelemetry Collector and Jaeger services to apply the changes.

By following these steps, you should be able to send trace data from OpenTelemetry to Jaeger in Kubernetes without encountering the "unknown service" error ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

The log in the jaeger-collector pod indicates that the gRPC server received an HTTP request instead of a gRPC request. This typically happens when the client is trying to connect to a gRPC endpoint using an HTTP protocol.

The specific log message:

grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""

This means that the gRPC server expected a gRPC request but received an HTTP request instead.

To resolve this issue, ensure that the OpenTelemetry Collector is configured to send gRPC requests to the Jaeger collector's gRPC endpoint. Verify that the configuration in your OpenTelemetry Collector's values.yaml file is correct:

exporters:
  otlp:
    endpoint: jaeger-collector.default.svc.cluster.local:4317
    tls:
      insecure: true

Make sure that the endpoint is set to the correct gRPC port (4317) and that the protocol used by the OpenTelemetry Collector matches the expected gRPC protocol.

Additionally, ensure that the Jaeger collector is configured to listen on the correct gRPC port:

collector:
  otlp:
    enabled: true
    grpc:
      host-port: ":4317"

By ensuring that both the OpenTelemetry Collector and Jaeger collector are correctly configured to use gRPC, you should be able to avoid this protocol mismatch and resolve the issue ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[yurishkuro]

14250 is grpc port for legacy Jaeger data model. If you are sending OTLP data then Jaeger is receiving it on the same port as OTEL collector.

[seikyo-cho-lvgs]

@yurishkuro
I installed jaeger and opentelemetry with the newest helm charts. The current jaeger service is using 14250 port.

kubectl get services
NAME                             TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                                   AGE
jaeger-agent                     ClusterIP      10.43.47.36     <none>         5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP             5d1h
jaeger-cassandra                 ClusterIP      None            <none>         7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP              5d1h
jaeger-collector                 ClusterIP      10.43.65.99     <none>         14250/TCP,14268/TCP,14269/TCP                             22h
jaeger-query                     ClusterIP      10.43.36.237    <none>         80/TCP,16685/TCP,16687/TCP                                5d1h
opentelemetry-collector          ClusterIP      10.43.188.170   <none>         6831/UDP,14250/TCP,14268/TCP,4317/TCP,4318/TCP,9411/TCP   47h
opentelemetry-operator           ClusterIP      10.43.237.13    <none>         8443/TCP,8080/TCP                                         5d3h
opentelemetry-operator-webhook   ClusterIP      10.43.51.200    <none>         443/TCP                                                   5d3h

If change to use 4317 port in opentelemery-collector's values

exporters:
  otlp:
    endpoint: jaeger-collector.default.svc.cluster.local:4317
    tls:
      insecure: true

The jaeger's service refused the request:

2024-07-23T07:18:08.671Z	warn	zapgrpc/zapgrpc.go:193	[core] [Channel #2 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: "10.43.209.8:4317", ServerName: "jaeger-collector.default.svc.cluster.local:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.43.209.8:4317: connect: connection refused"	{"grpc_log": true}
2024-07-23T07:18:15.120Z	info	exporterhelper/retry_sender.go:118	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.43.209.8:4317: connect: connection refused\"", "interval": "19.089395393s"}

I'm sure the jaeger values has been added:

helm get values jaeger
USER-SUPPLIED VALUES:
collector:
  otlp:
    enabled: true
    grpc:
      host-port: :4317

The opentelemetry-collector and jaeger service have been restarted.

[yurishkuro]

Jaeger enables OTLP ports 4317/4318 automatically since last year, unless explicitly disabled. But your mappings do not show that those ports are exposed in the container:

jaeger-collector                 ClusterIP      10.43.65.99     <none>         14250/TCP,14268/TCP,14269/TCP                             22h

[seikyo-cho-lvgs]

@yurishkuro
I don't know why those ports aren't been exported when I install by the latest helm chart:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger

Also jaeger-operator helm chart seems doesn't work, too:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger-operator jaegertracing/jaeger-operator

(I can't see simplest-collector service with many useful ports after create a Jaeger instance)

Finally this way works:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability

So is the issue the artifacthub.io helm charts are different from GitHub straightforward source?

[yurishkuro]

Please raise a ticket in the operator repo if you think there's a bug / mismatch. I am not an expert in how Helm versioning works, is it possible that when you say

helm install jaeger-operator jaegertracing/jaeger-operator

without specifying a version it might apply non-latest version (maybe there's a local cache)? This happens all the time with the Docker containers when you use :latest - it just defaults to what you have in your cache, rather than what upstream hub thinks is the latest.