Some nginx_ingress_controller_requests metrics do not contain ingress labels

[daviderli614]

What happened:
I queried the metrics of ingress nginx 4xx in the prometheus console and found that some metrics had the ingress label, while some did not.

nginx_ingress_controller_requests{namespace="ingress-nginx",status!="429",status=~"4.*",method="GET"}

What you expected to happen:
I don't understand why some metrics don't have the ingress label.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

kubectl exec -it ingress-nginx-controller-74696765cd-gd5vf /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
ingress-nginx-controller-74696765cd-gd5vf:/etc/nginx$ nginx-ingress-controller --version
bash: nginx-ingress-controller: command not found

Pod Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
Helm Chart Version: 4.8.3

Kubernetes version (use kubectl version):

kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6-eks-7f9249a

Environment:

• Cloud provider or hardware configuration:
  AWS EKS Cluster

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[daviderli614]

The helm chart configuration:

  controller:
    allowSnippetAnnotations: true
    replicaCount: 2
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 250m
        memory: 256Mi
    autoscaling:
      enabled: true
      annotations: {}
      minReplicas: 2
      maxReplicas: 5
      targetCPUUtilizationPercentage: 50
      targetMemoryUtilizationPercentage: 50
      behavior: {}
    ingressClassResource:
      enabled: true
      name: nginx
      controllerValue: "k8s.io/ingress-nginx"
    ingressClass: nginx
    extraArgs:
      metrics-per-host: false
    metrics:
      enabled: true
      serviceMonitor:
        enabled: true
        additionalLabels:
          release: prometheus
        namespace: "ingress-nginx"
        scrapeInterval: 10s
      prometheusRule:
        enabled: true
        namespace: "ingress-nginx"
        additionalLabels:
          release: prometheus
    config:
      use-gzip: "true"
      gzip-types: '*'
      proxy-body-size: "10m"
      error-log-level: info      

[longwuyuan]

/remove-kind bug

Computed metrics may not have labels

[daviderli614]

/remove-kind bug

Computed metrics may not have labels

If the request does not match the ingress, there is no label? How can I find these requests?