Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linkerd Tap doesn't seem to work with EKS Access Entries authentication #13169

Open
multimac opened this issue Oct 11, 2024 · 0 comments · May be fixed by #13170
Open

Linkerd Tap doesn't seem to work with EKS Access Entries authentication #13169

multimac opened this issue Oct 11, 2024 · 0 comments · May be fixed by #13170
Labels
bug env/eks Amazon EKS

Comments

@multimac
Copy link
Contributor

multimac commented Oct 11, 2024

What is the issue?

I was trying to use linkerd viz tap to debug an internal issue, but kept getting tap authorization failed. After looking at the documentation page in the error message (https://linkerd.io/2.16/tasks/securing-linkerd-tap/), it seemed like the problem might be in Linkerd itself as kubectl can-i works

How can it be reproduced?

I believe this might need an EKS cluster configured with EKS Access Entries for authentication :/

Logs, error output, etc

$ kubectl auth can-i watch deployments.v1alpha1.tap.linkerd.io/[xxx] --namespace [xxx] --subresource tap
yes
$ linkerd viz tap deployment/[xxx] --namespace [xxx]
HTTP error, status Code [403] (unexpected API response: {"error":"tap authorization failed (not authorized to access deployments.tap.linkerd.io), visit https://linkerd.io/tap-rbac for more information"})

output of linkerd check -o short

linkerd-identity
----------------
× trust anchors are using supported crypto algorithm
    Invalid trustAnchors:
        * 498364598153846236906649535816328601770266843336 One Model Linkerd Root CA must use P-256 curve for public key, instead P-384 was used
    see https://linkerd.io/2/checks/#l5d-identity-trustAnchors-use-supported-crypto for hints

linkerd-jaeger
--------------
‼ jaeger extension proxies are up-to-date
    some proxies are not running the current version:
        * collector-9cb9f47cb-2p4dn (edge-24.10.1)
        * collector-9cb9f47cb-s8zg2 (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-5xcmz (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-q46tg (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cp-version for hints
‼ jaeger extension proxies and cli versions match
    collector-9cb9f47cb-2p4dn running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cli-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * alertmanager-alertmanager-0 (edge-24.10.1)
        * metrics-api-fd554c577-8htqz (edge-24.10.1)
        * metrics-api-fd554c577-ftj2d (edge-24.10.1)
        * prometheus-prometheus-0 (edge-24.10.1)
        * tap-5579f7499d-2v4dg (edge-24.10.1)
        * tap-5579f7499d-5jf5j (edge-24.10.1)
        * tap-5579f7499d-bmdfg (edge-24.10.1)
        * tap-injector-5bfbc5b658-fqr7c (edge-24.10.1)
        * tap-injector-5bfbc5b658-kvz6s (edge-24.10.1)
        * tap-injector-5bfbc5b658-mrlmg (edge-24.10.1)
        * web-694c5b64d8-hs76l (edge-24.10.1)
        * web-694c5b64d8-kkqtp (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    alertmanager-alertmanager-0 running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints

Status check results are ×

Environment

Kubernetes Version: v1.30
Cluster Environment: AWS EKS
Host OS: Amazon Bottlerocket
Linkerd version: Client version: edge-24.10.2 / Server version: edge-24.10.1

Possible solution

I've made a commit which I will push up in a PR shortly, but I suspect the issue may be because the SubjectAccessReview done by the Tap controller doesn't pass in any of the "extra" user attributes

When looking at the audit logs generated by our Kubernetes control plane and comparing the linkerd viz tap vs. kubectl can-i, I can see that kubectl can-i is passing some additional "extra" fields that seem relevant to the EKS Access Entries authentication.

My commit updates ResourceAuthzForUser in linkerd/pkg/k8s/authz.go to take in the list of extra attributes, retrieved via the X-Remote-Extras- HTTP header

Additional context

No response

Would you like to work on fixing this bug?

yes

@multimac multimac added the bug label Oct 11, 2024
@olix0r olix0r added the env/eks Amazon EKS label Oct 11, 2024
multimac added a commit to multimac/linkerd2 that referenced this issue Oct 11, 2024
Kubernetes authorization plugins can rely on extra attributes on a user, and these are provided via `X-Remote-Extra-` headers. Currently the Linkerd Viz `tap` API doesn't include these attributes when making the `SubjectAccessReview` request which means the Tap API cannot be used by end-users who's clusters use such authz plugins.

This change updates the `tap` controller to parse the `X-Remote-Extra-` headers and include them in the SubjectAccessReview request.
Fixed linkerd#13169

Signed-off-by: David Symons <[email protected]>
@multimac multimac linked a pull request Oct 11, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug env/eks Amazon EKS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants