linkerd-proxy traces not showing up

[MrRexZ]

Hi everyone,

My linkerd server & client version is stable-2.9.1.
I'm trying to get linkerd proxy spans to show up in Jaeger by going through my custom opencensus collector deployment, but wasn't able to for unknown reason.
The following is the opencensus deployment manifest:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: oc-agent-conf
  labels:
    app: opencensus
    component: oc-agent-conf
data:
  oc-agent-config: |
    receivers:
      opencensus: {}
    exporters:
      opencensus:
        endpoint: "oc-collector.default:55678" # TODO: Update me
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: oc-agent
  labels:
    app: opencensus
    component: oc-agent
spec:
  selector:
    matchLabels:
      app: opencensus
      component: oc-agent
  template:
    metadata:
      labels:
        app: opencensus
        component: oc-agent
    spec:
      containers:
        - command:
            - "/ocagent_linux"
            - "--config=/conf/oc-agent-config.yaml"
          image: omnition/opencensus-agent:0.1.6
          name: oc-agent
          resources:
            limits:
              cpu: 500m
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 100Mi
          ports:
            - containerPort: 55678
            - containerPort: 55679
          #        - containerPort: 14267
          #        - containerPort: 14268
          #        - containerPort: 9411
          volumeMounts:
            - name: oc-agent-config-vol
              mountPath: /conf
      volumes:
        - configMap:
            name: oc-agent-conf
            items:
              - key: oc-agent-config
                path: oc-agent-config.yaml
          name: oc-agent-config-vol
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: oc-collector-conf
  labels:
    app: opencensus
    component: oc-collector-conf
data:
  oc-collector-config: |
    receivers:
      opencensus:
        keepalive:
          server-parameters:
            max-connection-age: 120s
            max-connection-age-grace: 30s
      jaeger: {}
      zipkin: {}
    exporters:
      jaeger:
        collector_endpoint: "http://jaeger-prod-collector.kube-system:14268/api/traces"
    queued-exporters:
      my-org-jaeger:
        num-workers: 2
        queue-size: 100
        retry-on-failure: true
        backoff-delay: 3s
        sender-type: jaeger-thrift-http
        jaeger-thrift-http:
          collector-endpoint: "http://jaeger-prod-collector.kube-system:14268/api/traces"
          timeout: 5s
---
apiVersion: v1
kind: Service
metadata:
  name: oc-collector
  labels:
    app: opencesus
    component: oc-collector
spec:
  ports:
    - name: opencensus
      port: 55678
      protocol: TCP
      targetPort: 55678
  #  - name: jaeger-tchannel
  #    port: 14267
  #  - name: jaeger-thrift-http
  #    port: 14268
  #  - name: zipkin
  #    port: 9411
  selector:
    component: oc-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oc-collector
  labels:
    app: opencensus
    component: oc-collector
spec:
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1
  selector:
    matchLabels:
      app: opencensus
      component: oc-collector
  template:
    metadata:
      annotations:
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8888"
        prometheus.io/scrape: "true"
      labels:
        app: opencensus
        component: oc-collector
    spec:
      containers:
        - command:
            - "/occollector_linux"
            - "--config=/conf/oc-collector-config.yaml"
          env:
            - name: GOGC
              value: "80"
          image: omnition/opencensus-collector:0.1.6
          name: oc-collector
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
          ports:
            - containerPort: 55678
          #        - containerPort: 14267
          #        - containerPort: 14268
          #        - containerPort: 9411
          volumeMounts:
            - name: oc-collector-config-vol
              mountPath: /conf
          #        - name: oc-collector-secrets
          #          mountPath: /secrets
          livenessProbe:
            httpGet:
              path: /
              port: 13133
          readinessProbe:
            httpGet:
              path: /
              port: 13133
      volumes:
        - configMap:
            name: oc-collector-conf
            items:
              - key: oc-collector-config
                path: oc-collector-config.yaml
          name: oc-collector-config-vol
#        - secret:
#            name: oc-collector-secrets
#            items:
#              - key: cert.pem
#                path: cert.pem
#              - key: key.pem
#                path: key.pem

I've added the following annotations to the service I want the proxy span to show up:

      linkerd.io/inject: enabled
      config.linkerd.io/trace-collector: oc-collector.default:55678

But still, none of the proxy spans showed up in my Jaeger UI. (I was able to see system-level traces (trace from jaeger itself), not from linkerd-proxy )

What I've done:

• Ensure both the custom opencensus collector and the service also already contains the proxy sidecar container
• Ensure there isn't any tracing component added in the application code, so there shouldn't be any problem about propagation format that may cause this issue.
• The open census collector is already using default service account.
• I've also looked at other threads with a similar issue : https://stackoverflow.com/questions/65377607/linkerd-distributed-tracing-with-opencensus , but was unable to find any answer/hint to this problem.
• I ran linkerd check command and everything is ok.
• Ensure that my jaeger is correctly set up (I was able to see my manually injected span, but not linkerd-proxy's spans)
• I checked linkerd-proxy in for one of my microservice instance, and no errors:

│ linkerd-proxy time="2021-01-04T19:46:05Z" level=info msg="running version stable-2.9.1"                                                                                        │
│ linkerd-proxy [     0.002410s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime                                                                      │
│ linkerd-proxy [     0.003574s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191                                                                              │
│ linkerd-proxy [     0.003647s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143                                                                            │
│ linkerd-proxy [     0.003670s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140                                                                         │
│ linkerd-proxy [     0.003689s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190                                                                                │
│ linkerd-proxy [     0.003729s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.default.serviceaccount.identity.linkerd.cluster.local                              │
│ linkerd-proxy [     0.003770s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.ser │
│ linkerd-proxy [     0.003792s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.s │
│ linkerd-proxy [     0.003811s]  INFO ThreadId(01) linkerd2_proxy: OpenCensus tracing collector at oc-collector.default:55678 (default.default.serviceaccount.identity.linkerd. │
│ linkerd-proxy [     0.020259s]  INFO ThreadId(01) outbound: linkerd2_app: listen.addr=127.0.0.1:4140 ingress_mode=false                                                        │
│ linkerd-proxy [     0.020643s]  INFO ThreadId(01) inbound: linkerd2_app: listen.addr=0.0.0.0:4143                                                                              │
│ linkerd-proxy [     0.399641s]  INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.default.serviceaccount.identity.linkerd.cluster.local

Is it because I need to use the custom linkerd-collector provided by linkerd instead of using my own custom one?
Any help would be appreciated, thanks a lot! :)

[MrRexZ]

I tried to use linkerd's open census collector, and by altering log levels, I'm getting the following logs in the linkerd-proxy:

2021-01-04T21:08:52.357090745Z [  1189.914768s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:34584 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=1162021-01-04T21:08:52.357116771Z [  1189.914797s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:34584 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::conditional_accept: match_client_hello: failed to parse up to SNI2021-01-04T21:08:52.357124276Z [  1189.914804s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:34584 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Not a matching TLS ClientHello
...
... Transport endpoint is not connected (os error 107)

Within these logs however, I can see the API calls are tracked:

2021-01-04T20:55:08.447027247Z [   366.004741s] TRACE ThreadId(01) outbound:tcp: linkerd2_duplex: writing 23B
2021-01-04T20:55:08.447297427Z [   366.004987s] TRACE ThreadId(01) outbound:tcp: linkerd2_duplex: wrote 23B
2021-01-04T20:55:08.447312371Z [   366.005005s] TRACE ThreadId(01) outbound:tcp: linkerd2_duplex: reading
2021-01-04T20:55:09.085269856Z [   366.642914s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd2_proxy_transport::listen: Accepted local.addr=10.32.0.7:4191 peer.addr=10.32.0.1:38540 orig.addr=None
2021-01-04T20:55:09.085494352Z [   366.643138s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38540 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=115
2021-01-04T20:55:09.085511182Z [   366.643163s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38540 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::conditional_accept: match_client_hello: failed to parse up to SNI
2021-01-04T20:55:09.085534981Z [   366.643170s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38540 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Not a matching TLS ClientHello
2021-01-04T20:55:09.085794963Z [   366.643494s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38540 target.addr=10.32.0.7:4191}: linkerd2_app_core::serve: Connection closed
2021-01-04T20:55:12.356976499Z [   369.914385s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd2_proxy_transport::listen: Accepted local.addr=10.32.0.7:4191 peer.addr=10.32.0.1:38630 orig.addr=None
2021-01-04T20:55:12.357014792Z [   369.914611s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38630 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=116
2021-01-04T20:55:12.357022114Z [   369.914633s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38630 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::conditional_accept: match_client_hello: failed to parse up to SNI
2021-01-04T20:55:12.357027983Z [   369.914639s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38630 target.addr=10.32.0.7:4191}: linkerd2_proxy_transport::tls::accept: Not a matching TLS ClientHello
2021-01-04T20:55:12.357134101Z [   369.914828s] DEBUG ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.1:38630 target.addr=10.32.0.7:4191}: linkerd2_app_core::serve: Connection closed
2021-01-04T20:55:13.457989116Z [   371.015515s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_metrics::serve: gzipping metrics
2021-01-04T20:55:13.45803423Z [   371.015585s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_http_metrics::requests::report: Formatting HTTP request metrics prefix="" targets=3 include_latencies=true
2021-01-04T20:55:13.45830265Z [   371.015986s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_http_metrics::requests::report: Formatting HTTP request metrics prefix="route" targets=2 include_latencies=true
2021-01-04T20:55:13.458671354Z [   371.016349s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_http_metrics::retries: Formatting HTTP retry metrics prfefix=route targets=0
2021-01-04T20:55:13.458689484Z [   371.016379s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_http_metrics::requests::report: Formatting HTTP request metrics prefix="route_actual" targets=2 include_latencies=false
2021-01-04T20:55:13.458695705Z [   371.016403s] TRACE ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.32.0.16:49120 target.addr=10.32.0.7:4191}: linkerd2_http_metrics::requests::report: Formatting HTTP request metrics prefix="control" targets=3 include_latencies=true
2021-01-04T20:55:17.151370184Z [   374.708998s] TRACE ThreadId(01) inbound: linkerd2_proxy_transport::listen: Accepted local.addr=10.32.0.7:4143 peer.addr=10.32.0.5:44034 orig.addr=Some(10.32.0.7:8083)
2021-01-04T20:55:17.170683962Z [   374.728318s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=512
2021-01-04T20:55:17.17071029Z [   374.728360s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::conditional_accept: match_client_hello: failed to parse up to SNI
2021-01-04T20:55:17.170717338Z [   374.728367s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::accept: Not a matching TLS ClientHello
2021-01-04T20:55:17.170723473Z [   374.728387s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_app_inbound::require_identity_for_ports: port=8083 peer.id=None(NoTlsFromRemote) id_required=false
2021-01-04T20:55:17.170729816Z [   374.728445s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::prefix: Buffering prefix capacity=8192
2021-01-04T20:55:17.170749077Z [   374.728489s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::prefix: read=2025
2021-01-04T20:55:17.170755569Z [   374.728519s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_buffer::dispatch: Dispatching request
2021-01-04T20:55:17.170761024Z [   374.728528s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_http::version: Detected H1
2021-01-04T20:55:17.170766822Z [   374.728534s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_http::detect: Handling as HTTP
2021-01-04T20:55:17.170772099Z [   374.728539s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}: linkerd2_proxy_http::detect: Building new HTTP service
2021-01-04T20:55:17.170807713Z [   374.728556s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_stack_tracing: new
2021-01-04T20:55:17.170953853Z [   374.728657s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_concurrency_limit: acquiring permit available=100000
2021-01-04T20:55:17.170966994Z [   374.728684s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_concurrency_limit: permit acquired available=99999
2021-01-04T20:55:17.170972478Z [   374.728691s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_stack_tracing: service ready=true ok=true
2021-01-04T20:55:17.171032324Z [   374.728738s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_stack_tracing: service request=Request { method: POST, uri: /jobs/overview-by-period, version: HTTP/1.1, headers: {"x-real-ip": "10.32.0.17", "host": "dtl.d", "x-forwarded-for": "10.32.0.1, 10.32.0.17", "accept-encoding": "gzip, deflate", "content-length": "35", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "content-type": "application/json;charset=utf-8", "cookie": "deploymentTimestamp=1609789503; session=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJSaWdodCBvZiBwYXNzYWdlIiwiZXhwIjoxNjExMDAxODk2LCJpYXQiOjE2MDk3OTIyOTYsInVzZXJJZCI6eyJpZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsIm5hbWUiOiJBbnRob255IFRqdWF0amEiLCJleHRJZCI6bnVsbCwiZXh0U291cmNlIjpudWxsfSwic2Vzc2lvbiI6eyJpZCI6ImJjMWM0M2Q4LWYzYTctNGI2Ny05OWUxLWFlMWEwNWVmZWE0MCIsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImNyZWF0ZWRBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgwWiIsImxhc3RBY3RpdmVBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgzWiIsImV4cGlyYXRpb25UaW1lb3V0SW5TZWNvbmRzIjoxNzI4MDB9fQ.bI_qhoCEDBmekwsiN-NKdkX8eT9OEDT8DnjUytus1pU; settings=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJMaWZlbG9uZyBzYXZpbmdzIiwiZXhwIjoxNjcyODY3ODk4LCJpYXQiOjE2MDk3OTIyOTgsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImN1ck9yZ0lkIjoiOGI3NWNkNTEtMDIzZS00MDMzLThiMWUtYTFlMjM0YWRlYmM5In0.fg2D6DtKaAnjtoV4pKGwW0ONaf5KAgAydaGgqgjy4Uo", "origin": "http://dtl.d", "referer": "http://dtl.d/apollo/", "x-forwarded-host": "dtl.d", "x-forwarded-port": "80", "x-forwarded-proto": "http", "x-forwarded-server": "traefik-ingress-controller-j28vj", "x_usr_id": "38765ee0-234e-4064-8c8e-ab28afa12815", "x_usr_org_id": "8b75cd51-023e-4033-8b1e-a1e234adebc9", "x_usr_group_ids": "668030bc-e91b-4be9-b625-0dc2a37906ec,dfcd906e-b679-4171-9115-d99586d9a5dc", "x_usr_orgs_ids": "[\"8b75cd51-023e-4033-8b1e-a1e234adebc9\"]", "x_usr_groups_ids": "[\"668030bc-e91b-4be9-b625-0dc2a37906ec\",\"dfcd906e-b679-4171-9115-d99586d9a5dc\"]", "x_org_admin": "true"}, body: Body { body: Some(Body(Streaming)), upgrade: None } }
2021-01-04T20:55:17.171055634Z [   374.728786s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_proxy_http::normalize_uri: Normalizing URI authority=dtl.d
2021-01-04T20:55:17.171061047Z [   374.728801s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}: linkerd2_app_core: l5d-dst-override not in request headers
2021-01-04T20:55:17.171079926Z [   374.728821s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_stack_tracing: new
2021-01-04T20:55:17.171097112Z [   374.728840s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_cache: Caching new service
2021-01-04T20:55:17.17112122Z [   374.728867s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_app_inbound::prevent_loop: addr=10.32.0.7:8083 self.port=4143
2021-01-04T20:55:17.171127808Z [   374.728885s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_stack_tracing: new
2021-01-04T20:55:17.171270731Z [   374.728981s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_cache: services=1 dropped=0
2021-01-04T20:55:17.171284223Z [   374.729035s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_stack_tracing: service ready=true ok=true
2021-01-04T20:55:17.171303959Z [   374.729044s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_stack_tracing: service request=Request { method: POST, uri: http://dtl.d/jobs/overview-by-period, version: HTTP/1.1, headers: {"x-real-ip": "10.32.0.17", "host": "dtl.d", "x-forwarded-for": "10.32.0.1, 10.32.0.17", "accept-encoding": "gzip, deflate", "content-length": "35", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "content-type": "application/json;charset=utf-8", "cookie": "deploymentTimestamp=1609789503; session=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJSaWdodCBvZiBwYXNzYWdlIiwiZXhwIjoxNjExMDAxODk2LCJpYXQiOjE2MDk3OTIyOTYsInVzZXJJZCI6eyJpZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsIm5hbWUiOiJBbnRob255IFRqdWF0amEiLCJleHRJZCI6bnVsbCwiZXh0U291cmNlIjpudWxsfSwic2Vzc2lvbiI6eyJpZCI6ImJjMWM0M2Q4LWYzYTctNGI2Ny05OWUxLWFlMWEwNWVmZWE0MCIsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImNyZWF0ZWRBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgwWiIsImxhc3RBY3RpdmVBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgzWiIsImV4cGlyYXRpb25UaW1lb3V0SW5TZWNvbmRzIjoxNzI4MDB9fQ.bI_qhoCEDBmekwsiN-NKdkX8eT9OEDT8DnjUytus1pU; settings=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJMaWZlbG9uZyBzYXZpbmdzIiwiZXhwIjoxNjcyODY3ODk4LCJpYXQiOjE2MDk3OTIyOTgsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImN1ck9yZ0lkIjoiOGI3NWNkNTEtMDIzZS00MDMzLThiMWUtYTFlMjM0YWRlYmM5In0.fg2D6DtKaAnjtoV4pKGwW0ONaf5KAgAydaGgqgjy4Uo", "origin": "http://dtl.d", "referer": "http://dtl.d/apollo/", "x-forwarded-host": "dtl.d", "x-forwarded-port": "80", "x-forwarded-proto": "http", "x-forwarded-server": "traefik-ingress-controller-j28vj", "x_usr_id": "38765ee0-234e-4064-8c8e-ab28afa12815", "x_usr_org_id": "8b75cd51-023e-4033-8b1e-a1e234adebc9", "x_usr_group_ids": "668030bc-e91b-4be9-b625-0dc2a37906ec,dfcd906e-b679-4171-9115-d99586d9a5dc", "x_usr_orgs_ids": "[\"8b75cd51-023e-4033-8b1e-a1e234adebc9\"]", "x_usr_groups_ids": "[\"668030bc-e91b-4be9-b625-0dc2a37906ec\",\"dfcd906e-b679-4171-9115-d99586d9a5dc\"]", "x_org_admin": "true"}, body: Payload }
2021-01-04T20:55:17.171364283Z [   374.729107s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_stack::switch_ready: SwitchReady::poll self.state=Primary
2021-01-04T20:55:17.171390083Z [   374.729132s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_service_profiles::default: Handling rejected discovery
2021-01-04T20:55:17.171415184Z [   374.729160s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_proxy_http::client: Building HTTP client settings=Http1
2021-01-04T20:55:17.171422112Z [   374.729176s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_stack_tracing: service ready=true ok=true
2021-01-04T20:55:17.171427567Z [   374.729185s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_buffer::dispatch: Dispatching request
2021-01-04T20:55:17.171433381Z [   374.729191s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}: linkerd2_stack::switch_ready: SwitchReady::call self.state=Primary
2021-01-04T20:55:17.171481654Z [   374.729198s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_stack_tracing: service request=Request { method: POST, uri: http://dtl.d/jobs/overview-by-period, version: HTTP/1.1, headers: {"x-real-ip": "10.32.0.17", "host": "dtl.d", "x-forwarded-for": "10.32.0.1, 10.32.0.17", "accept-encoding": "gzip, deflate", "content-length": "35", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "content-type": "application/json;charset=utf-8", "cookie": "deploymentTimestamp=1609789503; session=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJSaWdodCBvZiBwYXNzYWdlIiwiZXhwIjoxNjExMDAxODk2LCJpYXQiOjE2MDk3OTIyOTYsInVzZXJJZCI6eyJpZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsIm5hbWUiOiJBbnRob255IFRqdWF0amEiLCJleHRJZCI6bnVsbCwiZXh0U291cmNlIjpudWxsfSwic2Vzc2lvbiI6eyJpZCI6ImJjMWM0M2Q4LWYzYTctNGI2Ny05OWUxLWFlMWEwNWVmZWE0MCIsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImNyZWF0ZWRBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgwWiIsImxhc3RBY3RpdmVBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgzWiIsImV4cGlyYXRpb25UaW1lb3V0SW5TZWNvbmRzIjoxNzI4MDB9fQ.bI_qhoCEDBmekwsiN-NKdkX8eT9OEDT8DnjUytus1pU; settings=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJMaWZlbG9uZyBzYXZpbmdzIiwiZXhwIjoxNjcyODY3ODk4LCJpYXQiOjE2MDk3OTIyOTgsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImN1ck9yZ0lkIjoiOGI3NWNkNTEtMDIzZS00MDMzLThiMWUtYTFlMjM0YWRlYmM5In0.fg2D6DtKaAnjtoV4pKGwW0ONaf5KAgAydaGgqgjy4Uo", "origin": "http://dtl.d", "referer": "http://dtl.d/apollo/", "x-forwarded-host": "dtl.d", "x-forwarded-port": "80", "x-forwarded-proto": "http", "x-forwarded-server": "traefik-ingress-controller-j28vj", "x_usr_id": "38765ee0-234e-4064-8c8e-ab28afa12815", "x_usr_org_id": "8b75cd51-023e-4033-8b1e-a1e234adebc9", "x_usr_group_ids": "668030bc-e91b-4be9-b625-0dc2a37906ec,dfcd906e-b679-4171-9115-d99586d9a5dc", "x_usr_orgs_ids": "[\"8b75cd51-023e-4033-8b1e-a1e234adebc9\"]", "x_usr_groups_ids": "[\"668030bc-e91b-4be9-b625-0dc2a37906ec\",\"dfcd906e-b679-4171-9115-d99586d9a5dc\"]", "x_org_admin": "true"}, body: Payload }
2021-01-04T20:55:17.171504369Z [   374.729237s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile: linkerd2_service_profiles::http::route_request: Using default route
2021-01-04T20:55:17.171516713Z [   374.729257s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_http::client: method=POST uri=http://dtl.d/jobs/overview-by-period version=HTTP/1.1
2021-01-04T20:55:17.171553864Z [   374.729271s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_http::client: headers={"x-real-ip": "10.32.0.17", "host": "dtl.d", "x-forwarded-for": "10.32.0.1, 10.32.0.17", "accept-encoding": "gzip, deflate", "content-length": "35", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "content-type": "application/json;charset=utf-8", "cookie": "deploymentTimestamp=1609789503; session=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJSaWdodCBvZiBwYXNzYWdlIiwiZXhwIjoxNjExMDAxODk2LCJpYXQiOjE2MDk3OTIyOTYsInVzZXJJZCI6eyJpZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsIm5hbWUiOiJBbnRob255IFRqdWF0amEiLCJleHRJZCI6bnVsbCwiZXh0U291cmNlIjpudWxsfSwic2Vzc2lvbiI6eyJpZCI6ImJjMWM0M2Q4LWYzYTctNGI2Ny05OWUxLWFlMWEwNWVmZWE0MCIsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImNyZWF0ZWRBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgwWiIsImxhc3RBY3RpdmVBdCI6IjIwMjEtMDEtMDRUMjA6MzE6MzYuNzkzMzgzWiIsImV4cGlyYXRpb25UaW1lb3V0SW5TZWNvbmRzIjoxNzI4MDB9fQ.bI_qhoCEDBmekwsiN-NKdkX8eT9OEDT8DnjUytus1pU; settings=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJDZXJieSBvZiBEYXRhbG9ndWUiLCJzdWIiOiJMaWZlbG9uZyBzYXZpbmdzIiwiZXhwIjoxNjcyODY3ODk4LCJpYXQiOjE2MDk3OTIyOTgsInVzZXJJZCI6IjM4NzY1ZWUwLTIzNGUtNDA2NC04YzhlLWFiMjhhZmExMjgxNSIsImN1ck9yZ0lkIjoiOGI3NWNkNTEtMDIzZS00MDMzLThiMWUtYTFlMjM0YWRlYmM5In0.fg2D6DtKaAnjtoV4pKGwW0ONaf5KAgAydaGgqgjy4Uo", "origin": "http://dtl.d", "referer": "http://dtl.d/apollo/", "x-forwarded-host": "dtl.d", "x-forwarded-port": "80", "x-forwarded-proto": "http", "x-forwarded-server": "traefik-ingress-controller-j28vj", "x_usr_id": "38765ee0-234e-4064-8c8e-ab28afa12815", "x_usr_org_id": "8b75cd51-023e-4033-8b1e-a1e234adebc9", "x_usr_group_ids": "668030bc-e91b-4be9-b625-0dc2a37906ec,dfcd906e-b679-4171-9115-d99586d9a5dc", "x_usr_orgs_ids": "[\"8b75cd51-023e-4033-8b1e-a1e234adebc9\"]", "x_usr_groups_ids": "[\"668030bc-e91b-4be9-b625-0dc2a37906ec\",\"dfcd906e-b679-4171-9115-d99586d9a5dc\"]", "x_org_admin": "true"}
2021-01-04T20:55:17.171573269Z [   374.729312s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_http::h1: Using origin-form client
2021-01-04T20:55:17.171578917Z [   374.729319s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_http::h1: Caching new client use_absolute_form=false
2021-01-04T20:55:17.171646184Z [   374.729366s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_app_inbound::prevent_loop: addr=127.0.0.1:8083 self.port=4143
2021-01-04T20:55:17.171656287Z [   374.729388s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_transport::connect: Connecting peer.addr=127.0.0.1:8083
2021-01-04T20:55:17.171895994Z [   374.729586s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_transport::connect: Connected local.addr=127.0.0.1:38622 keepalive=None
2021-01-04T20:55:17.171921653Z [   374.729615s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44034 target.addr=10.32.0.7:8083}:http{v=1.x}:http1{name=dtl.d:80}:profile:http1: linkerd2_proxy_transport::metrics: client connection open
2021-01-04T20:55:17.226424303Z [   374.784049s] TRACE ThreadId(01) inbound: linkerd2_proxy_transport::listen: Accepted local.addr=10.32.0.7:4143 peer.addr=10.32.0.5:44040 orig.addr=Some(10.32.0.7:8083)
2021-01-04T20:55:17.245770109Z [   374.803437s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44040 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=512
2021-01-04T20:55:17.245803224Z [   374.803564s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44040 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::conditional_accept: match_client_hello: failed to parse up to SNI
2021-01-04T20:55:17.245885424Z [   374.803628s] TRACE ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44040 target.addr=10.32.0.7:8083}: linkerd2_proxy_transport::tls::accept: Not a matching TLS ClientHello
2021-01-04T20:55:17.24596483Z [   374.803710s] DEBUG ThreadId(01) inbound:accept{peer.addr=10.32.0.5:44040 target.addr=10.32.0.7:8083}: linkerd2_app_inbound::require_identity_for_ports: port=8083 peer.id=None(NoTlsFromRemote) id_required=false

Still, none of the trace is showing up in Jaeger UI.

[adleong]

Some part of your application (typically the ingress) will need to initiate the trace. See: https://linkerd.io/2/tasks/distributed-tracing/#ingress