Node NotReady after webhook-authenticator update

[joseluissmarting]

Hi all,

I have a 4.11.0-0.okd-2022-11-05-030711 cluster on baremetal and when the secret "webhook-authenticator" gets updated:
kind: ConfigMap
apiVersion: v1
metadata:
name: revision-status-37
namespace: openshift-kube-apiserver
managedFields:
- manager: cluster-kube-apiserver-operator
operation: Update
...
data:
reason: secret/webhook-authenticator has changed
revision: '37'

one master goes NotReady, kubelet service shows that the api don't recognize this node and can't connect. Then I have to reboot the server to connect it again to the cluster.

This happens almost all the times that this secret gets updated, each 10 days or so. It's strange that a couple of times, it worked fine.
I found that maybe it happens because the kube-apiserver pod doesn't start at right time, or the old one doesn't stop at right time..

It's possible to stop or going manual this updates?
or some sort of workaround for this, I don't know if it's some bug of the version, or any configuration that we have.

Kind regards,

Thank you in advance

[vrutkovs]

kubelet service shows that the api don't recognize this node and can't connect

what does it mean? Which "api" doesn't recognize the node?

It's possible to stop or going manual this updates?

You can disable the cluster-kube-apiserver operator, but that would make your certificates expire after ~month

[joseluissmarting]

It shows something like (systemctl status kubelet):
Error getting node X

Like if its doesn't exists, that node.
I think because the node still uses the old certificates and the apiserver is using the new one.

And, if we enable again the cluster-kube-apiserver operator, it would renew the certs again ?

[joseluissmarting]

And.. How can I disable/enable the kube-apiserver operator ?

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

Check if you have pending CSRs for nodes to be approved

[joseluissmarting]

I am having troubles to get the must-gather archive. I'll let you know if I get it

And no, no CSR is pending in this case, until the node reboot, it doesn't request a new certificate, CSR.

I will try to stop the kube-apiserver-operator, to set it unmanaged and then rollback, and see if it works, with:

oc edit kubeapiserver

[joseluissmarting]

I finally fix it, the solution was changing the dns order in /etc/nsswitch.conf.

from:
hosts: files myhostname resolve [!UNAVAIL=return] dns
to:
hosts: files myhostname resolve dns [!UNAVAIL=return]

then, the node resolve well the other nodes and api FQDN, and it doesn't get out of cluster.