How to expose dashboard with GCE ingress on GKE

[WLun001]

The ingress.yaml is configure as below

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: linkerd-web-ingress
  namespace: linkerd
  annotations:
  # with or without this custom-request-headers also not working
    ingress.kubernetes.io/custom-request-headers: "l5d-dst-override: linkerd-web.linkerd.svc.cluster.local:8084"
    kubernetes.io/ingress.global-static-ip-name: linkerd-ip
    networking.gke.io/managed-certificates: linkerd-certificate
spec:
  backend:
    serviceName: linkerd-web
    servicePort: 8084
  rules:
    - host: linkerd.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: linkerd-web
              servicePort: 8084

Error

jsonPayload: {
statusDetails: "failed_to_pick_backend"
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
}

Edited
linkerd-web is changed to NodePort instead of ClusterIP

[grampelberg]

It looks right to me... there might be a mismatch with the TTL? I'd verify that the linkerd-web pods are all healthy and reachable. You definitely don't need the header as the GCE ingress isn't injected with the proxy.

[WLun001]

Hi @grampelberg , i run linkerd dashboard locally, it works fine.
I also change linkerd-web to NodePort instead of ClusterIP, as GCE needs NodePort to work, but it still doesn't work

i tried port forward locally, it works too

kubectl port-forward service/linkerd-web 8084:8084 -n linkerd

[WLun001]

I guess something to do with readinessProbe? I checked the yaml of linkerd-web, it doesn't have readinessProbe
kubernetes/kubernetes#20555

But I have other deployment without readinessProbe, it still works fine with GCE ingress though.

[WLun001]

After investigate further, the error is INVALID_BACKEND, which is caused by failed health check

https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#filtering_dimensions_for_metrics

Not sure how to fix this

[grampelberg]

I'm not sure I have any great suggestions for you, make sure there are endpoints available in k8s. If there are, then there is an issue with the connection between the GCE load balancer and your k8s cluster.

[WLun001]

hmmmm. i have other GCE ingresses running on the same cluster, on default namespace, no issue.

[samstride]

@WLun001 , I am in the same situation. Looks like the GKE ingress requires an HTTP 200 returned at /.

Not sure how to solve that one.

[WLun001]

@samstride it should be up in few minutes
I think you need to remove the existing acme.json, and let it generate again
exec to the pod and remove it, should works

Remember to remove the line below when it works, to avoid hitting rate limit

--certificatesresolvers.myresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory

[samstride]

@WLun001 , since I am using helm, I uninstalled the chart and re-installed it again so that way acme.json is generated again.

I also added this line:

- --certificatesresolvers.myresolver.acme.dnsChallenge.resolvers=8.8.8.8

The errors are gone now. However, I cannot access linkerd.redacted.io and it's just blank. Hmmmm....

[WLun001]

@samstride have to set dnsChallenge.provider to gcloud if you were using Google Cloud DNS

[samstride]

@WLun001 , thanks. I finally managed to get it to work.

I am having a strange issue when installing the chart

• an external ip is automatically generated (as expected)
• it is automatically used in the traefik service (as expected)
• after about 30 seconds, the IP address is released

I have to manually apply it again to the traefik service and everything works.

Anyways, need to investigate more why I experience this.

[WLun001]

@samstride nice. Not sure what is the issue with the reserved IP address, let me know if you found the cause.

[tgadam]

After many machinations I was able to get this working using GCE. This requires multiple patches to the linkerd configuration. The issue is that GCE Ingress uses a default health check route which doesn't work with linkerd-web. The linkerd-web health check is at port 9994, path /ping.

GCE allows you to create a custom health check config. Now reading the GCE docs you may get confused like me and think you needed to add this BackendConfig to the Ingress yaml. (I'm using IAP for authentication to the ingress and that's how you configured it). However that's incorrect. You need to add the custom BackendConfig for the health check to the Service.

Once I figured that out I was able to get the linkerd-web service to return healthy! However that wasn't enough despite having:
ingress.kubernetes.io/custom-request-headers: "l5d-dst-override: linkerd-web.linkerd.svc.cluster.local:8084
in the ingress config, the header was not getting rewritten. So I ended patching the linkerd-web deployment to not do host checking (not the most secure but I'll try and patch it to the correct hostname eventually). After doing that it finally worked!

Please note there appears to some issues sometimes syncing changes depending on the order in which you apply changes (or it's possible I just wasn't patient enough for the changes to be scheduled and applied). I had to force update resources a couple of times before I finally got things working. Make sure and look at your events to ensure the changes actually got applied. Because I did so many iterations of this I'm unsure of the exact order to apply these. My guess based on GCP docs is that you should apply BackendConfigs first, then patches, and finally the ingress.

Below are my patch files and configs. I've omitted the certificate setup and IAP setup since those are well documented in the GCP docs:

Custom Health Check Config linkerd-web-service-bec.yaml

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: linkerd-web-healthcheck-config
  namespace: linkerd
spec:
  healthCheck:
    checkIntervalSec: 15
    timeoutSec: 10
    unhealthyThreshold: 5
    healthyThreshold: 1
    port: 9994
    type: HTTP
    requestPath: /ping

Here is the patch file for the linkerd-web Service linkerd-web-service-patch.yaml

metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "linkerd-web-healthcheck-config"}'
spec:
  type: NodePort

Patch Command (notice merge)

kubectl patch svc linkerd-web --type='merge' -p "$(cat linkerd-web-service-patch.yaml)" -n linkerd

Patch file for linkerd-web Deployment linkerd-web-deployment-patch.json

[
    {
        "op" : "replace",
        "path": "/spec/template/spec/containers/0/args/5",
        "value": "-enforced-host="
    }
]

Patch Command (notice json)

kubectl patch deployment linkerd-web --type='json' -p "$(cat linkerd-web-deployment-patch.json)" -n linkerd

Ingress Config (the backend config here is for IAP)

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: linkerd-dashboard
  namespace: linkerd
  annotations:
    # If the class annotation is not specified it defaults to "gce".
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: linkerd-static-ip
    networking.gke.io/managed-certificates: linkerd-dashboard-cert
    ingress.kubernetes.io/custom-request-headers: "l5d-dst-override: linkerd-web.linkerd.svc.cluster.local:8084" # doesn't seeem to work
    beta.cloud.google.com/backend-config: '{"default": "linkerd-dashboard-ingress-config"}'
spec:
  rules:
  - host: linkerd.teselagen.net
    http:
      paths:
      - backend:
          serviceName: linkerd-web
          servicePort: 8084

[samstride]

@tgadam , wow, thanks for taking the time and effort to solve the problem.

This has been my experience with GCE Ingress so far:

• Easy to setup
• Easy and automatic cert management with Google domains
• Works well for simple use cases

However, as soon as you start having many namespaces and you want to manage traffic routing for those namespaces, the simplest way is to get a new ingress for each namespace which tends to increase cost.

I am trying traefik based on @WLun001 feedback but I still can't get cert management working with google domains and I still can't access the dashboard.

Fun times :)

[WLun001]

@tgadam i tried this approach, but doesn't work, will try again

@samstride yes the 💰 is painful

[tgadam]

@WLun001 where did you run into issues?

[WLun001]

@tgadam It worked, I will mark this as answered. But I will be using Traefik. Thanks