custom-headers annotation doesn't work as expected

[satyamz]

What happened:
I have been trying to have custom response headers via annotation: nginx.ingress.kubernetes.io/custom-headers however, ingress endpoint is returning 503. Please find the relevant information below:

Ingress spec:

+ kubectl get ingress example-ingress -oyaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/custom-headers":"custom-headers"},"name":"example-ingress","namespace":"default"},"spec":{"ingressClassName":"nginx","rules":[{"host":"hello-world.example","http":{"paths":[{"backend":{"service":{"name":"web","port":{"number":8080}}},"path":"/","pathType":"Prefix"}]}}]}}
    nginx.ingress.kubernetes.io/custom-headers: custom-headers
  creationTimestamp: "2024-11-04T04:00:11Z"
  generation: 1
  name: example-ingress
  namespace: default
  resourceVersion: "1749"
  uid: ff46481c-36fd-46c0-b31c-e20d35788581
spec:
  ingressClassName: nginx
  rules:
  - host: hello-world.example
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 8080
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 192.168.49.2

ConfigMap:

+ kubectl get configmaps custom-headers -oyaml
apiVersion: v1
data:
  Name: satyam
  X-Different-Name: "true"
  X-Using-Nginx-Controller: "true"
kind: ConfigMap
metadata:
  creationTimestamp: "2024-11-04T04:08:28Z"
  name: custom-headers
  namespace: default
  resourceVersion: "1405"
  uid: b4edb759-b1cd-4c31-9481-9fc08d51f8df

Ingress controller configMap:

+ kubectl get configmap -n ingress-nginx ingress-nginx-controller
NAME                       DATA   AGE
ingress-nginx-controller   2      42m


+ kubectl get configmap -n ingress-nginx ingress-nginx-controller -oyaml
apiVersion: v1
data:
  global-allowed-response-headers: Name,X-Using-Nginx-Controller,X-Different-Name
  hsts: "false"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"hsts":"false"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"}}
  creationTimestamp: "2024-11-04T03:56:29Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.12.0-beta.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "1544"
  uid: e222f2e0-3273-48f8-9fb3-405ce32c4115

Ingress controller logs:

+ kubectl logs -f ingress-nginx-controller-6d5b999c7c-p8lgw -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.2
  Build:         46e76e5916813cfca2a9b0bfdc34b69a0000f6b9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

W1104 04:13:45.550296       7 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1104 04:13:45.550398       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I1104 04:13:45.553949       7 main.go:248] "Running in Kubernetes cluster" major="1" minor="25" git="v1.25.0" state="clean" commit="a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2" platform="linux/amd64"
I1104 04:13:45.841557       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1104 04:13:45.849740       7 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1104 04:13:45.855173       7 nginx.go:271] "Starting NGINX Ingress controller"
I1104 04:13:45.857733       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"e222f2e0-3273-48f8-9fb3-405ce32c4115", APIVersion:"v1", ResourceVersion:"1544", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1104 04:13:45.859969       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"28484d44-5c2b-4f1f-87d4-b8b2eb0ac641", APIVersion:"v1", ResourceVersion:"1178", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1104 04:13:45.859985       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"125131af-060c-436a-b68d-a32f98b25828", APIVersion:"v1", ResourceVersion:"1177", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1104 04:13:46.960950       7 store.go:440] "Found valid IngressClass" ingress="default/example-ingress" ingressclass="nginx"
I1104 04:13:46.961269       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"example-ingress", UID:"ff46481c-36fd-46c0-b31c-e20d35788581", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1690", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
E1104 04:13:46.961440       7 annotations.go:216] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"custom-headers\"" name="CustomHeaders" ingress="default/example-ingress"
I1104 04:13:47.057841       7 nginx.go:317] "Starting NGINX process"
I1104 04:13:47.057928       7 leaderelection.go:250] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I1104 04:13:47.058733       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1104 04:13:47.059338       7 controller.go:193] "Configuration changes detected, backend reload required"
I1104 04:13:47.067723       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-6d5b999c7c-bt82j"
I1104 04:13:47.120341       7 controller.go:213] "Backend successfully reloaded"
I1104 04:13:47.120408       7 controller.go:224] "Initial sync, sleeping for 1 second"
I1104 04:13:47.120457       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-6d5b999c7c-p8lgw", UID:"71b52b0a-b493-4d0e-9f48-a441f5193d9f", APIVersion:"v1", ResourceVersion:"1692", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

Specific log to notice in the above:

E1104 04:13:46.961440       7 annotations.go:216] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"custom-headers\"" name="CustomHeaders" ingress="default/example-ingress"

Curl to the endpoint:

satyamz@stellar upstream-ingress → curl --resolve "hello-world.example:80:$( minikube ip )" -i http://hello-world.example
HTTP/1.1 503 Service Temporarily Unavailable
Date: Mon, 04 Nov 2024 04:33:22 GMT
Content-Type: text/html
Content-Length: 190
Connection: keep-alive

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

What you expected to happen:
custom response headers should have been added to the response but application is failing and its returning 503.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v1.11.2

Kubernetes version (use kubectl version):
Tested on both 1.25 and 1.30
Environment:
EKS and minikube

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
    minikube and managed EKS.
• Basic cluster related info:
  • kubectl version

  + kubectl version
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.0

• kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
    minikube addon enable ingress

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

+ kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0-beta.0
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx
Events:       <none>


+ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-vbs8q        0/1     Completed   0          32m
ingress-nginx-admission-patch-fjqh7         0/1     Completed   0          32m
ingress-nginx-controller-6d5b999c7c-p8lgw   1/1     Running     0          24m

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

You can reproduce this issue in following steps:

1. Add config option global-allowed-response-headers with relevant header keys to ingress controller configMap.
2. Create configMap custom-headers-map containing headers in the same namespace as of ingress object.
3. Create ingress object with annotation: nginx.ingress.kubernetes.io/custom-headers: custom-headers-map
4. Curl the ingress endpoint and you should see 503s and controller logs showing configMap not found.

Anything else we need to know:
This issue is happening across all environments (Minikube, EKS, self hosted Kuberentes clusters). I tested it on 1.12 beta version of the ingress controller too.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug

I don't see there is a bug here.
I got the same error message about location denied.
Then I spec'd the cm name prefixed with the namespace like "default/$cmname" and i got a response code of 200

% helm -n ingress-nginx ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS      CHART                   APP VERSION
ingress-nginx   ingress-nginx   2               2024-11-04 12:04:04.995992157 +0530 IST deployed    ingress-nginx-4.11.3    1.11.3     
[~] 
% k describe cm customheaders 
Name:         customheaders
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
X-Using-Nginx-Controller:
----
true
Name:
----
Satyam
X-Different-Name:
----
true

BinaryData
====

Events:  <none>
[~] 
% k describe ingress httpd 
Name:             httpd
Labels:           <none>
Namespace:        default
Address:          192.168.58.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                Path  Backends
  ----                ----  --------
  httpd.mydomain.com  
                      /   httpd:80 (10.244.0.63:80)
Annotations:          nginx.ingress.kubernetes.io/custom-headers: default/customheaders
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    115s (x3 over 5m36s)  nginx-ingress-controller  Scheduled for sync
[~] 
% curl httpd.mydomain.com --resolve httpd.mydomain.com:80:`minikube ip` -i
HTTP/1.1 200 OK
Date: Mon, 04 Nov 2024 07:10:24 GMT
Content-Type: text/html
Content-Length: 45
Connection: keep-alive
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Name: Satyam
X-Different-Name: true
X-Using-Nginx-Controller: true

<html><body><h1>It works!</h1></body></html>

LOGS
% 
I1104 07:00:26.566752       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"                [25/63]
I1104 07:00:26.567020       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:00:26.568500       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-qbzrg"                                                                          
I1104 07:00:26.595566       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:00:26.595642       7 controller.go:224] "Initial sync, sleeping for 1 second"                                                                                                        
I1104 07:00:26.595678       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:01:21.199965       7 leaderelection.go:268] successfully acquired lease ingress-nginx/ingress-nginx-leader                                                                           
I1104 07:01:21.200015       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-kjhtj"                                                                          
E1104 07:01:24.676070       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.691561       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0.001
s admissionTime:0.016s testedConfigurationSize:20.0kB}                                                                                                                                        
I1104 07:01:24.691591       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:01:24.697160       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"8211227b-e7c7-4495-a6ea-b05552c037ac", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167125", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:01:24.697227       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.697483       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:01:24.725666       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:01:24.725816       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:03:01.198258       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:01.225498       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:03:01.225642       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
E1104 07:03:21.219327       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.234697       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.015s testedConfigurationSize:19.9kB}                                                                                                                                           
I1104 07:03:21.234725       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:03:21.238090       7 store.go:440] "Found valid IngressClass" ingress="default/httpd" ingressclass="nginx"                                                                           
E1104 07:03:21.238159       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.238188       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167241", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
I1104 07:03:21.238470       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:21.270151       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:03:21.270422       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:04:21.203640       7 status.go:304] "updating Ingress status" namespace="default" ingress="httpd" currentValue=null newValue=[{"ip":"192.168.58.2"}]                                 
I1104 07:04:21.208163       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167300", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:04:21.208326       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:04:21.208704       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:04:21.236874       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:04:21.237065       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:07:02.652708       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.017s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.017s testedConfigurationSize:22.1kB}                                             
I1104 07:07:02.652733       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"
I1104 07:07:02.656567       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167455", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1104 07:07:02.656841       7 controller.go:193] "Configuration changes detected, backend reload required"
I1104 07:07:02.684153       7 controller.go:213] "Backend successfully reloaded"               
I1104 07:07:02.684311       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.58.1 - - [04/Nov/2024:07:07:38 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 d4ceeb6e7508ab18b9bbdb7aef72c9d2
192.168.58.1 - - [04/Nov/2024:07:09:44 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.000 [default-httpd-80] [] 10.244.0.63:80 45 0.000 200 7750fde5babfc5f3dfef31f8047b381f
192.168.58.1 - - [04/Nov/2024:07:09:59 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 9e933118e104d67a7510d14cbbced3e2

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

/remove-kind bug

I don't there is a bug.
I got the same error message about location denied.
Then I spec'd the cm name prefixed with the namespace like "default/$cmname" and i got a response code of 200

% helm -n ingress-nginx ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS      CHART                   APP VERSION
ingress-nginx   ingress-nginx   2               2024-11-04 12:04:04.995992157 +0530 IST deployed    ingress-nginx-4.11.3    1.11.3     
[~] 
% k describe cm customheaders 
Name:         customheaders
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
X-Using-Nginx-Controller:
----
true
Name:
----
Satyam
X-Different-Name:
----
true

BinaryData
====

Events:  <none>
[~] 
% k describe ingress httpd 
Name:             httpd
Labels:           <none>
Namespace:        default
Address:          192.168.58.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
 Host                Path  Backends
 ----                ----  --------
 httpd.mydomain.com  
                     /   httpd:80 (10.244.0.63:80)
Annotations:          nginx.ingress.kubernetes.io/custom-headers: default/customheaders
Events:
 Type    Reason  Age                   From                      Message
 ----    ------  ----                  ----                      -------
 Normal  Sync    115s (x3 over 5m36s)  nginx-ingress-controller  Scheduled for sync
[~] 
% curl httpd.mydomain.com --resolve httpd.mydomain.com:80:`minikube ip` -i
HTTP/1.1 200 OK
Date: Mon, 04 Nov 2024 07:10:24 GMT
Content-Type: text/html
Content-Length: 45
Connection: keep-alive
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Name: Satyam
X-Different-Name: true
X-Using-Nginx-Controller: true

<html><body><h1>It works!</h1></body></html>

LOGS
% 
I1104 07:00:26.566752       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"                [25/63]
I1104 07:00:26.567020       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:00:26.568500       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-qbzrg"                                                                          
I1104 07:00:26.595566       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:00:26.595642       7 controller.go:224] "Initial sync, sleeping for 1 second"                                                                                                        
I1104 07:00:26.595678       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:01:21.199965       7 leaderelection.go:268] successfully acquired lease ingress-nginx/ingress-nginx-leader                                                                           
I1104 07:01:21.200015       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-kjhtj"                                                                          
E1104 07:01:24.676070       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.691561       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0.001
s admissionTime:0.016s testedConfigurationSize:20.0kB}                                                                                                                                        
I1104 07:01:24.691591       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:01:24.697160       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"8211227b-e7c7-4495-a6ea-b05552c037ac", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167125", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:01:24.697227       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.697483       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:01:24.725666       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:01:24.725816       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:03:01.198258       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:01.225498       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:03:01.225642       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
E1104 07:03:21.219327       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.234697       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.015s testedConfigurationSize:19.9kB}                                                                                                                                           
I1104 07:03:21.234725       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:03:21.238090       7 store.go:440] "Found valid IngressClass" ingress="default/httpd" ingressclass="nginx"                                                                           
E1104 07:03:21.238159       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.238188       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167241", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
I1104 07:03:21.238470       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:21.270151       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:03:21.270422       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:04:21.203640       7 status.go:304] "updating Ingress status" namespace="default" ingress="httpd" currentValue=null newValue=[{"ip":"192.168.58.2"}]                                 
I1104 07:04:21.208163       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167300", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:04:21.208326       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:04:21.208704       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:04:21.236874       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:04:21.237065       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:07:02.652708       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.017s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.017s testedConfigurationSize:22.1kB}                                             
I1104 07:07:02.652733       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"
I1104 07:07:02.656567       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167455", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1104 07:07:02.656841       7 controller.go:193] "Configuration changes detected, backend reload required"
I1104 07:07:02.684153       7 controller.go:213] "Backend successfully reloaded"               
I1104 07:07:02.684311       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.58.1 - - [04/Nov/2024:07:07:38 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 d4ceeb6e7508ab18b9bbdb7aef72c9d2
192.168.58.1 - - [04/Nov/2024:07:09:44 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.000 [default-httpd-80] [] 10.244.0.63:80 45 0.000 200 7750fde5babfc5f3dfef31f8047b381f
192.168.58.1 - - [04/Nov/2024:07:09:59 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 9e933118e104d67a7510d14cbbced3e2

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[satyamz]

Thanks for triaging this quickly. Let me open a PR to update documentation. Its what landed me in this situation:

---

[longwuyuan]

@satyamz Thanks in advance for fixing docs. If you can additionally avoid names like "custom-headers" for the configMap (and also address other such aesthetics, if any required), it could be an improvement as well.

Secondly, kindly review the e2e-tests for this and see if they can be improved too (cosmetic and otherwise). Regards

[longwuyuan]

duplicate of #11339