kubelet stopped posting node status

[imdmahajankanika]

Describe the bug

Hello! We tried to upgrade our OKD cluster from 4.10->4.11 (4.11.0-0.okd-2022-07-29-154152, UPI Vsphere), the upgrade went well on master nodes, all the master nodes got successfully upgraded to FCOS 36, v1.24.0. But as soon as it started on worker nodes, the first worker node got updated to FCOS 36 but stayed in state "NotReady, schedulingDisabled".We tried to login to the respective worker node and checked kubelet status, it showed

root@worker-0 core]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-mco-default-madv.conf, 20-logging.conf, 20-nodenet.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2023-08-02 21:26:31 UTC; 157ms ago
    Process: 6723 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
    Process: 6724 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
    Process: 6725 ExecStartPre=/bin/rm -f /var/lib/kubelet/memory_manager_state (code=exited, status=0/SUCCESS)
    Process: 6726 ExecStart=/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtim>
   Main PID: 6726 (code=exited, status=1/FAILURE)
        CPU: 76ms

And, with the command journalctl -b -f -u kubelet.service, it has shown below error

Aug 02 20:23:50 [worker-0.outcastv2-preprod.davidson-idf.fr](http://worker-0.outcastv2-preprod.davidson-idf.fr/) hyperkube[16974]: Error: failed to set feature gates from initial flags-based config: cannot set feature gate RemoveSelfLink to false, feature is locked to true
Aug 02 20:23:50 [worker-0.outcastv2-preprod.davidson-idf.fr](http://worker-0.outcastv2-preprod.davidson-idf.fr/) hyperkube[16974]: Usage:

Apart from that, there were no errors observed. Even the machineConfigPool was stuck in status "Updating" not "degraded". Also, I was able to oc login from that node.
And the result of rpm-ostree status was below:-

State: idle
Deployments:
● pivot://quay.io/openshift/okd-content@sha256:a77c75b002aa480b9dc834a8c0cb38ccce347f528f1d27118c08da2bb2e199b1
             CustomOrigin: Managed by machine-config-operator
                  Version: 411.36.202207291018-0 (2022-07-29T10:22:03Z)  pivot://quay.io/openshift/okd-content@sha256:cd4fe63cf7c5c787dfe4f0080630459c20a7b87ce8754415a2277d934563ec66
             CustomOrigin: Managed by machine-config-operator
                  Version: 410.35.202206101257-0 (2022-06-10T13:01:20Z)

Version

4.11.0-0.okd-2022-07-29-154152, UPI Vsphere
How reproducible

During the OKD upgrade from 4.10.0-0.okd-2022-06-10-131327 to 4.11.0-0.okd-2022-07-29-154152

Log bundle

Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --register-node                                            Register the node with the apiserver. If --kubeconfig is not >
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --register-schedulable                                     Register the node as schedulable. Won't have any effect if re>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --register-with-taints []v1.Taint                          Register the node with the given list of taints (comma separa>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --registry-burst int32                                     Maximum size of a bursty pulls, temporarily allows pulls to b>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --registry-qps int32                                       If > 0, limit registry pull QPS to this value.  If 0, unlimit>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --reserved-cpus string                                     A comma-separated list of CPUs or CPU ranges that are reserve>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --reserved-memory reserved-memory                          A comma separated list of memory reservations for NUMA nodes.>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --resolv-conf string                                       Resolver configuration file used as the basis for the contain>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --root-dir string                                          Directory path for managing kubelet files (volume mounts,etc)>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --rotate-certificates                                      <Warning: Beta feature> Auto rotate the kubelet client certif>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --rotate-server-certificates                               Auto-request and rotate the kubelet serving certificates by r>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --runonce                                                  If true, exit after spawning pods from static pod files or re>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --runtime-cgroups string                                   Optional absolute name of cgroups to create and run the runti>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --runtime-request-timeout duration                         Timeout of all runtime requests except long running request ->
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --seccomp-default RuntimeDefault                           <Warning: Alpha feature> Enable the use of RuntimeDefault as >
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --serialize-image-pulls                                    Pull images one at a time. We recommend *not* changing the de>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --skip-headers                                             If true, avoid header prefixes in the log messages (DEPRECATE>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --skip-log-headers                                         If true, avoid headers when opening log files (DEPRECATED: wi>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --stderrthreshold severity                                 logs at or above this threshold go to stderr (default 2) (DEP>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-buffer-duration duration                  Writes in the storage driver will be buffered for this durati>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-db string                                 database name (default "cadvisor") (DEPRECATED: This is a cad>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-host string                               database host:port (default "localhost:8086") (DEPRECATED: Th>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-password string                           database password (default "root") (DEPRECATED: This is a cad>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-secure                                    use secure connection with database (DEPRECATED: This is a ca>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-table string                              table name (default "stats") (DEPRECATED: This is a cadvisor >
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --storage-driver-user string                               database username (default "root") (DEPRECATED: This is a cad>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --streaming-connection-idle-timeout duration               Maximum time a streaming connection can be idle before the co>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --sync-frequency duration                                  Max period between synchronizing running containers and confi>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --system-cgroups string                                    Optional absolute name of cgroups in which to place all non-k>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --system-reserved mapStringString                          A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --system-reserved-cgroup string                            Absolute name of the top level cgroup that is used to manage >
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --tls-cert-file string                                     File containing x509 Certificate used for serving HTTPS (with>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --tls-cipher-suites strings                                Comma-separated list of cipher suites for the server. If omit>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:                 Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDS>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:                 Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --tls-min-version string                                   Minimum TLS version supported. Possible values: VersionTLS10,>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --tls-private-key-file string                              File containing x509 private key matching --tls-cert-file. (D>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --topology-manager-policy string                           Topology Manager policy to use. Possible values: 'none', 'bes>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --topology-manager-scope string                            Scope to which topology hints applied. Topology Manager colle>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:   -v, --v Level                                                  number for the log level verbosity (default 0)
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --version version[=true]                                   Print version information and quit
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --vmodule pattern=N,...                                    comma-separated list of pattern=N settings for file-filtered >
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --volume-plugin-dir string                                 The full path of the directory in which to search for additio>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr hyperkube[7976]:       --volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the vol>
Aug 02 20:40:43 worker-0.outcastv2-preprod.davidson-idf.fr systemd[1]: Failed to start kubelet.service - Kubernetes Kubelet.
░░ Subject: A start job for unit kubelet.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit kubelet.service has finished with a failure.
░░ 
░░ The job identifier is 6779 and the job result is failed.

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[imdmahajankanika]

@vrutkovs I am sorry, we had to rollback the cluster as it was the pre-production cluster. So I won't be able to provide the logs.

Thanks for your help!

[imdmahajankanika]

@vrutkovs I found the root cause, it happened only for workers because kubelet config (machine config generated) of worker nodes was different than that of masters and there was a feature gate RemoveSelfLink in the workers' kubelet config which was set to false and was causing the problem.

Error: failed to set feature gates from initial flags-based config: cannot set feature gate RemoveSelfLink to false, feature is locked to true