v1.12.0 seemingly breaks ingresses with auth snippets

[craigcabrey]

What happened:

Upgrade to v1.12.0 using Helm chart 4.12.0 (I do this using Argo if it matters). Many ingresses which make use of a forward auth seem to break with a 404.

What you expected to happen:

Existing ingresses continue to function as expected.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

before (working):

internal-ingress-nginx-controller-6d5597c8dc-2q2dv:/etc/nginx$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.4
  Build:         ba0f2ee37f032c9f11967b74862c60a43ed59b36
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

after (broken):

internal-ingress-nginx-controller-6bd59b7898-zgrs4:/etc/nginx$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.0
  Build:         ba73b2c24d355f1cdcf4b31ef7c5574059f12118
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Ingress with auth snippets applied (e.g. for Authentik integration):

  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: https://auth.[snip]/outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-authentik-embedded-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx

Kubernetes version (use kubectl version): v1.29.12+k3s1

Environment:

• Cloud provider or hardware configuration: Self hosted homelab

• OS (e.g. from /etc/os-release): Fedora CoreOS 41.20241109.3.0

• Kernel (e.g. uname -a): 6.11.6-300.fc41.x86_64

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed: Helm via Argo (not sure how to get the full release out of argo)

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:         external
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=external
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.11.3
              argocd.argoproj.io/instance=external-ingress
              helm.sh/chart=ingress-nginx-4.11.3
Annotations:  <none>
Controller:   k8s.io/external-ingress-nginx
Events:       <none>


Name:         internal
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=internal
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0
              argocd.argoproj.io/instance=internal-ingress
              helm.sh/chart=ingress-nginx-4.12.0
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/internal-ingress-nginx
Events:       <none>

Logs show it sees the ingress:

I0110 05:00:54.802592       7 store.go:440] "Found valid IngressClass" ingress="teslamate/prod-teslamate" ingressclass="internal"
I0110 05:00:54.802724       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"teslamate", Name:"prod-teslamate", UID:"f3ecd5ad-fa14-4a8e-808b-ebd93978f530", APIVersion:"networking.k8s.io/v1", ResourceVersion:"145859890", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0110 05:00:54.802799       7 backend_ssl.go:67] "Adding secret to local store" name="teslamate/teslamate.[snip]-prod-tls"

Anything else we need to know:

Reliably reproducible. No difference in Helm chart versions, I isolated it down to the v1.12.0 image of the nginx controller causing the unexpected behavior.

Ingresses wihtout auth snippets function as expected.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[abhiverma001]

@craigcabrey I also faced the same issue For this issue now we need to set controller.config.annotations-risk-level: Critical in order to use snippets due to breaking changes in a minor version bump..

The "risk" list is here.. Annotations Risks - Ingress-Nginx Controller

[Socolin]

I got the same problem, invalid TLS certificate after upgrading

@craigcabrey I also faced the same issue For this issue now we need to set controller.config.annotations-risk-level: Critical in order to use snippets due to breaking changes in a minor version bump..

This fixed the problem, Thanks !

I was not expecting a breaking change on minor version bump too.

[craigcabrey]

thanks, can confirm this also works for me. I'll leave this open for the maintainers to decide what to do with it.