Linkerd breaks k8s service contract

[Facecube]

Linkerd breaks the k8s service contract when routing meshed requests, targeting k8s services.

This discussion is only valid for old releases.

Found in version: 2.14.5
Fixed in version: I don't know exactly. For sure in 2.19.

Consider the following (simplified) scenario:

One would expect the request to fail, because the k8s service does not define port 80:

In reality however, the outbound linkerd proxy forwards the request, disregarding the k8s service contract.
In my understanding the components involved in handling the outgoing request interact in the following simplified manner:

Debugging this problem further led me to the linkerd diagnostics cli.
The diagnostics cli verifies that linkerd indeed resolves any random port for a service name.

$ kubectl get svc -n emojivoto
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
emoji-svc    ClusterIP   10.43.232.154   <none>        8080/TCP,8801/TCP   34m
voting-svc   ClusterIP   10.43.24.83     <none>        8080/TCP,8801/TCP   34m
web-svc      ClusterIP   10.43.18.80     <none>        80/TCP              34m

$ ./linkerd2-cli-stable-2.14.5-darwin-arm64 diagnostics endpoints web-svc.emojivoto.svc.cluster.local:80
NAMESPACE   IP           PORT   POD                    SERVICE
            10.42.0.54   80     web-55ff647458-cwrhv   .

$ ./linkerd2-cli-stable-2.14.5-darwin-arm64 diagnostics endpoints web-svc.emojivoto.svc.cluster.local:1234
NAMESPACE   IP           PORT   POD                    SERVICE
emojivoto   10.42.0.54   1234   web-55ff647458-cwrhv   web-svc.emojivoto

$ ./linkerd2-cli-stable-2.14.5-darwin-arm64 diagnostics endpoints web-svc.emojivoto.svc.cluster.local:6942
NAMESPACE   IP           PORT   POD                    SERVICE
emojivoto   10.42.0.54   6942   web-55ff647458-cwrhv   web-svc.emojivoto

The experiment can be reproduced by deploying the emojivoto demo application and modifying its web-svc service.
In combination with nginx's default server semantics, this can lead to very strange unexpected behavior (requests being successfully handled which should fail).

Documentation about how linkerd destination resolution works is rather sparse.
However, doing the same experiment with 2.19 shows that this "bug" has been fixed.

I did not find a github issue concerning itself with this problem.
Maybe someone knowledgeable in the community can link the respective PR that fixed this and save others running on "legacy" versions some time and their sanity.

[bezarsnba]

Hi, Are you using transport-header or transparent? The recent version linkerd default is transport-header: https://linkerd.buoyant.io/t/clarification-on-outbound-transport-mode/768/2

[Facecube]

I was using default values for the transport mode, for the respective linkerd versions (seems like they may have changed between the respective versions).

I am not very familiar with the meshed proxy->proxy connection.
However I think that the problem occurs rather a bit earlier, in the destination lookup, as demonstrated by the linkerd diagnostics cli experiment.

I will still give modifying the transport mode a chance on newer versions a chance, when I have time, to see if I can then also reproduce the problem there too.

Thanks for the reply!