Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when installing ingress controller with helm #12130

Closed
aolloh opened this issue Oct 7, 2024 · 21 comments
Closed

Error when installing ingress controller with helm #12130

aolloh opened this issue Oct 7, 2024 · 21 comments
Labels
kind/support Categorizes issue or PR as a support question. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@aolloh
Copy link

aolloh commented Oct 7, 2024

Hello,
Any advice for this issue will be welcomed. Thank you in advance.

[vagrant@k8s-ctrl-1 ~]$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
 
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition

Logs and environment info

[vagrant@k8s-ctrl-1 ~]$ helm version
version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"clean", GoVersion:"go1.22.7"}

[vagrant@k8s-ctrl-1 ~]$ free -th
               total        used        free      shared  buff/cache   available
Mem:           3.5Gi       1.2Gi       866Mi        10Mi       1.7Gi       2.3Gi
Swap:             0B          0B          0B
Total:         3.5Gi       1.2Gi       866Mi

[vagrant@k8s-ctrl-1 ~]$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
CPU(s):                  4
...

# Kube nodes

[vagrant@k8s-ctrl-1 ~]$ kubectl get nodes
NAME                         STATUS   ROLES           AGE   VERSION
k8s-ctrl-1.example.local     Ready    control-plane   12d   v1.31.1
k8s-worker-1.example.local   Ready    <none>          12d   v1.31.1
k8s-worker-2.example.local   Ready    <none>          12d   v1.31.1

# Kube namespaces

[vagrant@k8s-ctrl-1 ~]$ kubectl get ns
NAME              STATUS   AGE
default           Active   12d
ingress-nginx     Active   74m
kube-node-lease   Active   12d
kube-public       Active   12d
kube-system       Active   12d

# Ingress controller status

[vagrant@k8s-ctrl-1 ~]$ kubectl get jobs -n ingress-nginx
NAME                             STATUS   COMPLETIONS   DURATION   AGE
ingress-nginx-admission-create   Failed   0/1           12m        12m

[vagrant@k8s-ctrl-1 ~]$ kubectl get svc -n ingress-nginx
No resources found in ingress-nginx namespace.

[vagrant@k8s-ctrl-1 ~]$ kubectl get pods -n ingress-nginx
No resources found in ingress-nginx namespace.

[vagrant@k8s-ctrl-1 ~]$ helm ls -A | grep -i ingress
ingress-nginx                  	ingress-nginx	3       	2024-10-07 10:36:38.961355306 +0000 UTC	failed  	ingress-nginx-4.11.2                  	1.11.2     

[vagrant@k8s-ctrl-1 ~]$ kubectl describe jobs ingress-nginx-admission-create -n ingress-nginx
Name:             ingress-nginx-admission-create
Namespace:        ingress-nginx
Selector:         batch.kubernetes.io/controller-uid=5b8478ff-4645-450e-a0f8-187ac6373fc5
Labels:           app.kubernetes.io/component=admission-webhook
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.11.2
                  helm.sh/chart=ingress-nginx-4.11.2
Annotations:      helm.sh/hook: pre-install,pre-upgrade
                  helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
Parallelism:      1
Completions:      1
Completion Mode:  NonIndexed
Suspend:          false
Backoff Limit:    6
Start Time:       Mon, 07 Oct 2024 10:36:40 +0000
Pods Statuses:    0 Active (0 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=admission-webhook
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.11.2
                    batch.kubernetes.io/controller-uid=5b8478ff-4645-450e-a0f8-187ac6373fc5
                    batch.kubernetes.io/job-name=ingress-nginx-admission-create
                    controller-uid=5b8478ff-4645-450e-a0f8-187ac6373fc5
                    helm.sh/chart=ingress-nginx-4.11.2
                    job-name=ingress-nginx-admission-create
  Service Account:  ingress-nginx-admission
  Containers:
   create:
    Image:           registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    Environment:
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
  Node-Selectors:     kubernetes.io/os=linux
  Tolerations:        <none>
Events:
  Type     Reason                Age    From            Message
  ----     ------                ----   ----            -------
  Normal   SuccessfulCreate      15m    job-controller  Created pod: ingress-nginx-admission-create-c5gwz
  Normal   SuccessfulDelete      6m36s  job-controller  Deleted pod: ingress-nginx-admission-create-c5gwz
  Warning  BackoffLimitExceeded  6m35s  job-controller  Job has reached the specified backoff limit


[vagrant@k8s-ctrl-1 ~]$ kubectl edit job ingress-nginx-admission-create -n ingress-nginx
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  creationTimestamp: "2024-10-07T10:36:40Z"
  generation: 1
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.2
    helm.sh/chart: ingress-nginx-4.11.2
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
  resourceVersion: "334900"
  uid: 5b8478ff-4645-450e-a0f8-187ac6373fc5
spec:
  backoffLimit: 6
  completionMode: NonIndexed
  completions: 1
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: 5b8478ff-4645-450e-a0f8-187ac6373fc5
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.11.2
        ....
status:
  conditions:
  - lastProbeTime: "2024-10-07T10:45:23Z"
    lastTransitionTime: "2024-10-07T10:45:23Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: FailureTarget
  - lastProbeTime: "2024-10-07T10:45:24Z"
    lastTransitionTime: "2024-10-07T10:45:24Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: Failed
  failed: 1
  ready: 0
  startTime: "2024-10-07T10:36:40Z"
  terminating: 0
  uncountedTerminatedPods: {}
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Oct 7, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority labels Oct 7, 2024
@Gacko
Copy link
Member

Gacko commented Oct 7, 2024

Hey,

it's pretty clear the job failing. Unfortunately the job status doesn't help a lot. Can you maybe re-install the chart and check why the job pods are failing? Some log output or pod status would be helpful.

Regards
Marco

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

Able to install ingress when I disable webhook but ingress controller doesn't work

[vagrant@k8s-ctrl-1 ~]$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace --set controller.admissionWebhooks.enabled=false

Release "ingress-nginx" does not exist. Installing it now.
NAME: ingress-nginx
LAST DEPLOYED: Wed Oct  9 10:43:34 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the load balancer IP to be available.
You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Pod logs

[vagrant@k8s-ctrl-1 ~]$ kubectl get pods -n ingress-nginx
NAME                                       READY   STATUS             RESTARTS     AGE
ingress-nginx-controller-b954bfd68-pshgb   0/1     CrashLoopBackOff   5 (2s ago)   5m23s

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                           READY   STATUS             RESTARTS      AGE
pod/ingress-nginx-controller-b954bfd68-pshgb   0/1     CrashLoopBackOff   5 (21s ago)   5m42s

NAME                               TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller   LoadBalancer   10.96.8.7    <pending>     80:31637/TCP,443:30202/TCP   5m42s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   0/1     1            0           5m42s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-b954bfd68   1         1         0       5m42s


[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-controller-b954bfd68-pshgb -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.3
  Build:         0106de65cfccb74405a6dfa7d9daffc6f0a6ef1a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

W1009 10:46:26.473204       7 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1009 10:46:26.473464       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"

[vagrant@k8s-ctrl-1 ~]$ kubectl describe pod ingress-nginx-controller-b954bfd68-pshgb -n ingress-nginx
Name:             ingress-nginx-controller-b954bfd68-pshgb
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             k8s-worker-2.example.local/192.168.178.164
Start Time:       Wed, 09 Oct 2024 10:43:35 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.11.3
                  helm.sh/chart=ingress-nginx-4.11.3
                  pod-template-hash=b954bfd68
Annotations:      cni.projectcalico.org/containerID: 31abf82075e099fe913fd67cf8f85f90af41336e63e57b20e9bee16713769a93
                  cni.projectcalico.org/podIP: 192.168.243.164/32
                  cni.projectcalico.org/podIPs: 192.168.243.164/32
Status:           Running
IP:               192.168.243.164
IPs:
  IP:           192.168.243.164
Controlled By:  ReplicaSet/ingress-nginx-controller-b954bfd68
Containers:
  controller:
    Container ID:    containerd://d745eaf7bf2f57c6391061f18eed095634ad4f2bcedeffe76db4b7df262bed9f
    Image:           registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
    Ports:           80/TCP, 443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --enable-metrics=false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Wed, 09 Oct 2024 10:48:06 +0000
      Finished:     Wed, 09 Oct 2024 10:48:56 +0000
    Ready:          False
    Restart Count:  5
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-b954bfd68-pshgb (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fxqmg (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-fxqmg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason             Age                    From               Message
  ----     ------             ----                   ----               -------
  Normal   Scheduled          6m32s                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-b954bfd68-pshgb to k8s-worker-2.example.local
  Normal   Pulling            6m32s                  kubelet            Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7"
  Normal   Pulled             6m19s                  kubelet            Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 12.831s (12.831s including waiting). Image size: 104685884 bytes.
  Normal   Created            5m22s (x2 over 6m19s)  kubelet            Created container controller
  Normal   Started            5m22s (x2 over 6m19s)  kubelet            Started container controller
  Normal   Killing            5m22s                  kubelet            Container controller failed liveness probe, will be restarted
  Warning  FailedPreStopHook  5m22s                  kubelet            PreStopHook failed
  Normal   Pulled             5m22s                  kubelet            Container image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" already present on machine
  Warning  Unhealthy          5m2s (x8 over 6m2s)    kubelet            Readiness probe failed: Get "http://192.168.243.164:10254/healthz": dial tcp 192.168.243.164:10254: connect: connection refused
  Warning  Unhealthy          82s (x29 over 6m2s)    kubelet            Liveness probe failed: Get "http://192.168.243.164:10254/healthz": dial tcp 192.168.243.164:10254: connect: connection refused


[vagrant@k8s-ctrl-1 ~]$ sudo firewall-cmd --list-all | grep 10254
  ports: 6443/tcp 2379/tcp 2380/tcp 10250/tcp 10251/tcp 10252/tcp 10257/tcp 10259/tcp 179/tcp 4789/udp 2049/tcp 111/tcp 10254/tcp
  
 [vagrant@k8s-ctrl-1 ~]$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.11.3
                          helm.sh/chart=ingress-nginx-4.11.3
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: ingress-nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.8.7
IPs:                      10.96.8.7
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31637/TCP
Endpoints:                192.168.243.164:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30202/TCP
Endpoints:                192.168.243.164:443
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:                   <none> 

[vagrant@k8s-ctrl-1 ~]$ kubectl describe deploy ingress-nginx-controller -n ingress-nginx
Name:                   ingress-nginx-controller
Namespace:              ingress-nginx
CreationTimestamp:      Wed, 09 Oct 2024 10:43:35 +0000
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.11.3
                        helm.sh/chart=ingress-nginx-4.11.3
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: ingress-nginx
                        meta.helm.sh/release-namespace: ingress-nginx
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.11.3
                    helm.sh/chart=ingress-nginx-4.11.3
  Service Account:  ingress-nginx
  Containers:
   controller:
    Image:           registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
    Ports:           80/TCP, 443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --enable-metrics=false
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:           <none>
  Volumes:            <none>
  Node-Selectors:     kubernetes.io/os=linux
  Tolerations:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   ingress-nginx-controller-b954bfd68 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  9m6s  deployment-controller  Scaled up replica set ingress-nginx-controller-b954bfd68 to 1

@Gacko
Copy link
Member

Gacko commented Oct 9, 2024 •
edited
Loading

Please do not turn off the webhook but rather gather some details on the failing webhook certificate creation & patch job pods.

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

I deleted the ingress namespace and start a new installation. Still have the same issue

[vagrant@k8s-ctrl-1 ~]$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
Release "ingress-nginx" does not exist. Installing it now.
Error: failed pre-install: 1 error occurred:
	* timed out waiting for the condition
  • Checking the progress on another session
[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-admission-create-bxvc9   1/1     Running   0          9s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           9s         9s
  • Logs ( the pod is unable to reach the Kubernetes API server (at 10.96.0.1:443))
[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-admission-create-bxvc9 -n ingress-nginx
W1009 11:24:34.874773       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"err":"Get \"https://10.96.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission\": dial tcp 10.96.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2024-10-09T11:25:04Z"}

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS   RESTARTS      AGE
pod/ingress-nginx-admission-create-bxvc9   0/1     Error    2 (55s ago)   2m

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           2m         2m

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx

NAME                                       READY   STATUS             RESTARTS      AGE
pod/ingress-nginx-admission-create-bxvc9   0/1     CrashLoopBackOff   3 (39s ago)   3m25s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           3m25s      3m25s

[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-admission-create-bxvc9 -n ingress-nginx

W1009 11:27:27.823595       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  • Describe pod
[vagrant@k8s-ctrl-1 ~]$ kubectl describe pod ingress-nginx-admission-create-bxvc9 -n ingress-nginx
Name:             ingress-nginx-admission-create-bxvc9
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx-admission
Node:             k8s-worker-1.example.local/192.168.178.163
Start Time:       Wed, 09 Oct 2024 11:23:59 +0000
Labels:           app.kubernetes.io/component=admission-webhook
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.11.3
                  batch.kubernetes.io/controller-uid=02a69045-284e-48e5-acca-0fe1ce3e2e6c
                  batch.kubernetes.io/job-name=ingress-nginx-admission-create
                  controller-uid=02a69045-284e-48e5-acca-0fe1ce3e2e6c
                  helm.sh/chart=ingress-nginx-4.11.3
                  job-name=ingress-nginx-admission-create
Annotations:      cni.projectcalico.org/containerID: 8b4f16913d53a21e1d99e51b720ab8ec1612e57aca254622c3f01dda1263e254
                  cni.projectcalico.org/podIP: 192.168.106.160/32
                  cni.projectcalico.org/podIPs: 192.168.106.160/32
Status:           Running
IP:               192.168.106.160
IPs:
  IP:           192.168.106.160
Controlled By:  Job/ingress-nginx-admission-create
Containers:
  create:
    Container ID:    containerd://d0de40c63225407ca03d5b3c39239adf7dc0f0a02d0456c0076951ad8e116d02
    Image:           registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
    Image ID:        registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 09 Oct 2024 11:27:27 +0000
      Finished:     Wed, 09 Oct 2024 11:27:57 +0000
    Ready:          False
    Restart Count:  4
    Environment:
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5r476 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-5r476:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  4m42s                default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-bxvc9 to k8s-worker-1.example.local
  Normal   Pulling    4m42s                kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f"
  Normal   Pulled     4m39s                kubelet            Successfully pulled image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" in 3.739s (3.739s including waiting). Image size: 26152957 bytes.
  Normal   Created    75s (x5 over 4m39s)  kubelet            Created container create
  Normal   Started    75s (x5 over 4m38s)  kubelet            Started container create
  Normal   Pulled     75s (x4 over 4m8s)   kubelet            Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" already present on machine
  Warning  BackOff    18s (x9 over 3m37s)  kubelet            Back-off restarting failed container create in pod ingress-nginx-admission-create-bxvc9_ingress-nginx(0338ba0c-b121-4867-97f4-e07823b1a071)
  • Other info
[vagrant@k8s-ctrl-1 ~]$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS      AGE
coredns-7c65d6cfc9-mzsfc   1/1     Running   9 (52m ago)   14d
coredns-7c65d6cfc9-zlfhb   1/1     Running   9 (52m ago)   14d


## pod/ingress-nginx-admission-create-bxvc9   disappears alone

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       STATUS   COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Failed   0/1           9m8s       9m8s


[vagrant@k8s-ctrl-1 ~]$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   14d

[vagrant@k8s-ctrl-1 ~]$ kubectl get endpoints -n default kubernetes
NAME         ENDPOINTS              AGE
kubernetes   192.168.178.162:6443   14d

@longwuyuan
Copy link
Contributor

Open ports 8443 and 10254 between all nodes inside the cluster

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

  • Ports opened but same issue
[vagrant@k8s-ctrl-1 ~]$  sudo firewall-cmd --list-ports
111/tcp 179/tcp 2049/tcp 2379/tcp 2380/tcp 6443/tcp 8443/tcp 10250/tcp 10251/tcp 10252/tcp 10254/tcp 10257/tcp 10259/tcp 4789/udp

[vagrant@k8s-worker-1 ~]$ sudo firewall-cmd --list-ports
179/tcp 8443/tcp 10250/tcp 10254/tcp 30000-32767/tcp 4789/udp

[vagrant@k8s-worker-2 ~]$ sudo firewall-cmd --list-ports
179/tcp 8443/tcp 10250/tcp 10254/tcp 30000-32767/tcp 4789/udp

## Installation

[vagrant@k8s-ctrl-1 ~]$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
Release "ingress-nginx" does not exist. Installing it now.
Error: failed pre-install: 1 error occurred:
	* timed out waiting for the condition

## Checking

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS   RESTARTS      AGE
pod/ingress-nginx-admission-create-ww7xp   0/1     Error    1 (38s ago)   69s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           70s        70s


[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS    RESTARTS      AGE
pod/ingress-nginx-admission-create-ww7xp   1/1     Running   2 (21s ago)   83s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           83s        83s

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS    RESTARTS      AGE
pod/ingress-nginx-admission-create-ww7xp   1/1     Running   4 (44s ago)   3m26s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           3m26s      3m26s

[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-admission-create-ww7xp -n ingress-nginx
W1009 12:07:36.854907       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS             RESTARTS      AGE
pod/ingress-nginx-admission-create-ww7xp   0/1     CrashLoopBackOff   4 (30s ago)   4m24s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           4m24s      4m24s

[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-admission-create-ww7xp -n ingress-nginx

W1009 12:07:36.854907       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"err":"Get \"https://10.96.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission\": dial tcp 10.96.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2024-10-09T12:08:06Z"}
  • Describe job
[vagrant@k8s-ctrl-1 ~]$ kubectl describe job ingress-nginx-admission-create -n ingress-nginx
Name:             ingress-nginx-admission-create
Namespace:        ingress-nginx
Selector:         batch.kubernetes.io/controller-uid=5b2ecfb5-4411-43a2-84f5-883c46cfbedb
Labels:           app.kubernetes.io/component=admission-webhook
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.11.3
                  helm.sh/chart=ingress-nginx-4.11.3
Annotations:      helm.sh/hook: pre-install,pre-upgrade
                  helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
Parallelism:      1
Completions:      1
Completion Mode:  NonIndexed
Suspend:          false
Backoff Limit:    6
Start Time:       Wed, 09 Oct 2024 12:04:12 +0000
Pods Statuses:    1 Active (0 Ready) / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=admission-webhook
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.11.3
                    batch.kubernetes.io/controller-uid=5b2ecfb5-4411-43a2-84f5-883c46cfbedb
                    batch.kubernetes.io/job-name=ingress-nginx-admission-create
                    controller-uid=5b2ecfb5-4411-43a2-84f5-883c46cfbedb
                    helm.sh/chart=ingress-nginx-4.11.3
                    job-name=ingress-nginx-admission-create
  Service Account:  ingress-nginx-admission
  Containers:
   create:
    Image:           registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    Environment:
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
  Node-Selectors:     kubernetes.io/os=linux
  Tolerations:        <none>
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  6m44s  job-controller  Created pod: ingress-nginx-admission-create-ww7xp
  • log
[vagrant@k8s-ctrl-1 ~]$ kubectl describe pod ingress-nginx-admission-create-ww7xp -n ingress-nginx
Name:             ingress-nginx-admission-create-ww7xp
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx-admission
Node:             k8s-worker-1.example.local/192.168.178.163
Start Time:       Wed, 09 Oct 2024 12:04:12 +0000
Labels:           app.kubernetes.io/component=admission-webhook
                 app.kubernetes.io/instance=ingress-nginx
                 app.kubernetes.io/managed-by=Helm
                 app.kubernetes.io/name=ingress-nginx
                 app.kubernetes.io/part-of=ingress-nginx
                 app.kubernetes.io/version=1.11.3
                 batch.kubernetes.io/controller-uid=5b2ecfb5-4411-43a2-84f5-883c46cfbedb
                 batch.kubernetes.io/job-name=ingress-nginx-admission-create
                 controller-uid=5b2ecfb5-4411-43a2-84f5-883c46cfbedb
                 helm.sh/chart=ingress-nginx-4.11.3
                 job-name=ingress-nginx-admission-create
Annotations:      cni.projectcalico.org/containerID: 2482944c546afad98c26192ae9875a33840047d8bc90d538bbeee3889b5314c5
                 cni.projectcalico.org/podIP: 192.168.106.161/32
                 cni.projectcalico.org/podIPs: 192.168.106.161/32
Status:           Running
IP:               192.168.106.161
IPs:
 IP:           192.168.106.161
Controlled By:  Job/ingress-nginx-admission-create
Containers:
 create:
   Container ID:    containerd://98b03a674328931204ed44bfe993ebbacc6601c7ab1495a36aeb0d22ab9e1173
   Image:           registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
   Image ID:        registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
   Port:            <none>
   Host Port:       <none>
   SeccompProfile:  RuntimeDefault
   Args:
     create
     --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
     --namespace=$(POD_NAMESPACE)
     --secret-name=ingress-nginx-admission
   State:          Waiting
     Reason:       CrashLoopBackOff
   Last State:     Terminated
     Reason:       Error
     Exit Code:    1
     Started:      Wed, 09 Oct 2024 12:09:34 +0000
     Finished:     Wed, 09 Oct 2024 12:10:04 +0000
   Ready:          False
   Restart Count:  5
   Environment:
     POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
   Mounts:
     /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cqpjd (ro)
Conditions:
 Type                        Status
 PodReadyToStartContainers   True 
 Initialized                 True 
 Ready                       False 
 ContainersReady             False 
 PodScheduled                True 
Volumes:
 kube-api-access-cqpjd:
   Type:                    Projected (a volume that contains injected data from multiple sources)
   TokenExpirationSeconds:  3607
   ConfigMapName:           kube-root-ca.crt
   ConfigMapOptional:       <nil>
   DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                            node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
 Type     Reason     Age                    From               Message
 ----     ------     ----                   ----               -------
 Normal   Scheduled  6m52s                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-ww7xp to k8s-worker-1.example.local
 Normal   Pulled     3m29s (x5 over 6m52s)  kubelet            Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" already present on machine
 Normal   Created    3m29s (x5 over 6m52s)  kubelet            Created container create
 Normal   Started    3m29s (x5 over 6m52s)  kubelet            Started container create
 Warning  BackOff    106s (x12 over 5m50s)  kubelet            Back-off restarting failed container create in pod ingress-nginx-admission-create-ww7xp_ingress-nginx(95c7a97b-62b5-4b8c-b13a-c5588aba37e6)

@longwuyuan
Copy link
Contributor

That is not proof.

Disable firewalld just to test.
Also use the --debug flag of helm.
And before you try install each time, You need to uninstall and make sure that no objects & resources created by previous installs is still lingering.

Without log messages of helm install with --debug, its hard to tell what the problem is.

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

  • Yes, I always uninstall the old one before trying a new installation
$ helm uninstall ingress-nginx --namespace ingress-nginx
$ kubectl delete ns ingress-nginx
  • I disabled the firewall on all nodes
  • Reinstallation with --debug flag
[vagrant@k8s-ctrl-1 ~]$ helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace --debug
history.go:56: 2024-10-09 12:42:20.340749926 +0000 UTC m=+0.077885992 [debug] getting history for release ingress-nginx
Release "ingress-nginx" does not exist. Installing it now.
install.go:224: 2024-10-09 12:42:20.344012081 +0000 UTC m=+0.081148138 [debug] Original chart version: ""
install.go:241: 2024-10-09 12:42:20.963435192 +0000 UTC m=+0.700571326 [debug] CHART PATH: /home/vagrant/.cache/helm/repository/ingress-nginx-4.11.3.tgz

client.go:142: 2024-10-09 12:42:21.640432197 +0000 UTC m=+1.377568270 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:21.697606967 +0000 UTC m=+1.434743020 [debug] Starting delete for "ingress-nginx-admission" ServiceAccount
client.go:490: 2024-10-09 12:42:21.699892833 +0000 UTC m=+1.437028899 [debug] Ignoring delete failure for "ingress-nginx-admission" /v1, Kind=ServiceAccount: serviceaccounts "ingress-nginx-admission" not found
wait.go:104: 2024-10-09 12:42:21.699992333 +0000 UTC m=+1.437128395 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:21.839321633 +0000 UTC m=+1.576457693 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:21.849425692 +0000 UTC m=+1.586561748 [debug] Starting delete for "ingress-nginx-admission" ClusterRole
wait.go:104: 2024-10-09 12:42:21.853075375 +0000 UTC m=+1.590211427 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:21.924012706 +0000 UTC m=+1.661148757 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:21.933348582 +0000 UTC m=+1.670484634 [debug] Starting delete for "ingress-nginx-admission" ClusterRoleBinding
wait.go:104: 2024-10-09 12:42:21.936671004 +0000 UTC m=+1.673807056 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:22.015547458 +0000 UTC m=+1.752683517 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:22.024965114 +0000 UTC m=+1.762101179 [debug] Starting delete for "ingress-nginx-admission" Role
client.go:490: 2024-10-09 12:42:22.026769939 +0000 UTC m=+1.763905996 [debug] Ignoring delete failure for "ingress-nginx-admission" rbac.authorization.k8s.io/v1, Kind=Role: roles.rbac.authorization.k8s.io "ingress-nginx-admission" not found
wait.go:104: 2024-10-09 12:42:22.026811028 +0000 UTC m=+1.763947085 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:22.099179912 +0000 UTC m=+1.836315975 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:22.107317721 +0000 UTC m=+1.844453786 [debug] Starting delete for "ingress-nginx-admission" RoleBinding
client.go:490: 2024-10-09 12:42:22.10964634 +0000 UTC m=+1.846782404 [debug] Ignoring delete failure for "ingress-nginx-admission" rbac.authorization.k8s.io/v1, Kind=RoleBinding: rolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" not found
wait.go:104: 2024-10-09 12:42:22.109718655 +0000 UTC m=+1.846854713 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:22.183736722 +0000 UTC m=+1.920872784 [debug] creating 1 resource(s)
client.go:486: 2024-10-09 12:42:22.193878905 +0000 UTC m=+1.931014961 [debug] Starting delete for "ingress-nginx-admission-create" Job
client.go:490: 2024-10-09 12:42:22.195938206 +0000 UTC m=+1.933074262 [debug] Ignoring delete failure for "ingress-nginx-admission-create" batch/v1, Kind=Job: jobs.batch "ingress-nginx-admission-create" not found
wait.go:104: 2024-10-09 12:42:22.19598287 +0000 UTC m=+1.933118924 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2024-10-09 12:42:22.277218497 +0000 UTC m=+2.014354559 [debug] creating 1 resource(s)
client.go:712: 2024-10-09 12:42:22.283661756 +0000 UTC m=+2.020797814 [debug] Watching for changes to Job ingress-nginx-admission-create with timeout of 5m0s
client.go:740: 2024-10-09 12:42:22.312673888 +0000 UTC m=+2.049809985 [debug] Add/Modify event for ingress-nginx-admission-create: ADDED
client.go:779: 2024-10-09 12:42:22.312719595 +0000 UTC m=+2.049855654 [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:42:22.313280678 +0000 UTC m=+2.050416748 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:42:22.313305034 +0000 UTC m=+2.050441093 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:42:24.724459375 +0000 UTC m=+4.461595492 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:42:24.724566104 +0000 UTC m=+4.461702193 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:42:54.960684612 +0000 UTC m=+34.697820681 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:42:54.960722977 +0000 UTC m=+34.697859037 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:42:55.980416977 +0000 UTC m=+35.717553104 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:42:55.980520281 +0000 UTC m=+35.717656380 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:43:26.204069604 +0000 UTC m=+65.941205710 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:43:26.204125007 +0000 UTC m=+65.941261086 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:43:39.575411711 +0000 UTC m=+79.312547812 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:43:39.575483039 +0000 UTC m=+79.312619121 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:44:10.523849343 +0000 UTC m=+110.260985410 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:44:10.523876502 +0000 UTC m=+110.261012560 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:44:36.7229254 +0000 UTC m=+136.460061486 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:44:36.723010327 +0000 UTC m=+136.460146411 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:45:07.066199608 +0000 UTC m=+166.803335740 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:45:07.066318061 +0000 UTC m=+166.803454159 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:45:58.407787512 +0000 UTC m=+218.144923689 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:45:58.407944305 +0000 UTC m=+218.145080436 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: 2024-10-09 12:46:28.802441792 +0000 UTC m=+248.539577927 [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:779: 2024-10-09 12:46:28.802693913 +0000 UTC m=+248.539830021 [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: failed pre-install: 1 error occurred:
	* timed out waiting for the condition


helm.go:86: 2024-10-09 12:47:22.356683392 +0000 UTC m=+302.093819448 [debug] failed pre-install: 1 error occurred:
	* timed out waiting for the condition
  • error
[vagrant@k8s-ctrl-1 ~]$ kubectl get all -n ingress-nginx
NAME                                       READY   STATUS             RESTARTS      AGE
pod/ingress-nginx-admission-create-m5bxp   0/1     CrashLoopBackOff   5 (33s ago)   6m37s

NAME                                       STATUS    COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Running   0/1           6m37s      6m37s

[vagrant@k8s-ctrl-1 ~]$ kubectl logs ingress-nginx-admission-create-m5bxp -n ingress-nginx

W1009 12:45:56.775936       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"err":"Get \"https://10.96.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission\": dial tcp 10.96.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2024-10-09T12:46:26Z"}

@longwuyuan
Copy link
Contributor

what is the result of this ;

echo "                       
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-2-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.11.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx-2
    app.kubernetes.io/version: "1.11.3"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
automountServiceAccountToken: true
" | kubectl create -f -


kubectl get sa

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024 

[vagrant@k8s-ctrl-1 ~]$ kubectl get sa
NAME                              SECRETS   AGE
default                           0         14d
nfs-subdir-external-provisioner   0         5d19h

@longwuyuan
Copy link
Contributor

You are not able to create serviceAccount so its not a problem with the controller.

Don't use vagrant. Just use minikube or kind.

@longwuyuan
Copy link
Contributor

longwuyuan commented Oct 9, 2024 •
edited
Loading

Also check the logs for

echo "                       
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-2-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-4.11.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx-2
    app.kubernetes.io/version: "1.11.3"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
automountServiceAccountToken: true
" | kubectl create -f -  --v=8


kubectl get sa -A | grep -i admission

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024 

[vagrant@k8s-ctrl-1 ~]$ kubectl get sa -A | grep -i admission
ingress-nginx     ingress-nginx-admission                       0         54m
kube-system       validatingadmissionpolicy-status-controller   0         14d

@longwuyuan
Copy link
Contributor

Yeah, you are not able to create serviceAccount so its not a problem with the controller.

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

You are not able to create serviceAccount so its not a problem with the controller.

Don't use vagrant. Just use minikube or kind.

Yes, very easy in minikube

$ minikube addons enable ingress

@longwuyuan
Copy link
Contributor

ok, please close the issue, since the problem is solved.

@aolloh
Copy link
Author

aolloh commented Oct 9, 2024

No, the problem is not solved
I said it's easy to enable ingress on minikube.

@longwuyuan
Copy link
Contributor

longwuyuan commented Oct 9, 2024 via email

@longwuyuan
Copy link
Contributor

/kind support

All information provided here shows that serviceaccount can not be created by the user. SA creation fails both manually using a kubectl create -f from a manifest and obviously while installaing the helm chart.

So this is not a problem of the ingress-nginx controller. Hence closing this issue as theree is no action item here for the project to track.

@aolloh its best that you engage discussion on the Kubernetes slack, where there may be other users of the same environment as yours. If you later find a bug in the controller code, then post the details of that here and re-open the issue.

/close

@k8s-ci-robot k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Oct 10, 2024
@k8s-ci-robot
Copy link
Contributor

@longwuyuan: Closing this issue.

In response to this:

/kind support

All information provided here shows that serviceaccount can not be created by the user. SA creation fails both manually using a kubectl create -f from a manifest and obviously while installaing the helm chart.

So this is not a problem of the ingress-nginx controller. Hence closing this issue as theree is no action item here for the project to track.

@aolloh its best that you engage discussion on the Kubernetes slack, where there may be other users of the same environment as yours. If you later find a bug in the controller code, then post the details of that here and re-open the issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support Categorizes issue or PR as a support question. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
[SIG Network] Ingress NGINX
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Gacko @longwuyuan @k8s-ci-robot @aolloh