Intermittent node NotReady Because kubelet not able to update the status to kube-apiserver

[balait4]

After upgrade to OKD version 4.10 (4.10.0-0.okd-2022-06-10-131327) we seeing below issue only on few worker nodes.
This issue is intermittent. During this time node become NotReady and after around 1 hour time its back to normal.
Even we upgraded to latest version 4.12(4.12.0-0.okd-2023-04-01-051724) and still facing the same issue.
Also the node lease url seems to be invalid https://api-init/./? Any thought on this please.

Log from worker node:
=====================
 kubenswrapper[1384]: E0504 07:22:55.493751    1384 controller.go:187] failed to update lease, error:
Put "https://api-int/./apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/?timeout=10s": net/http:
request canceled (Client.Timeout exceeded while awaiting headers)

 kubenswrapper[1384]: E0504 07:22:55.954915    1384 kubelet_node_status.go:487] "Error updating node status, will retry" err="error getting
 node \"\": Get \"https://api-int/.:/api/v1/nodes/?resourceVersion=0&timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

Log from Kube-apiserver: (at the same time)
=======================
E0504 07:30:40.919989      15 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated, token lookup failed]"
E0504 07:30:41.764021      15 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated, token lookup failed]"
E0504 07:30:42.794170      15 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated, token lookup failed]"
E0504 07:30:43.871626      15 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated, token lookup failed]"

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[balait4]

Here is the must-gather logs.
https://drive.google.com/file/d/1aunBHSVYfq1EviK5ii2sdQmYWrgsiM33/view?usp=share_link

[vrutkovs]

I don't see bad api-int URL anywhere in the logs. Which worker was that?

As for the not ready nodes I only see master has been not ready:

NodeControllerDegraded: The master nodes not ready: node "mst-2-nbg-eu.k8sprod.targomo.com" not ready since 2023-04-29 14:09:22 +0000 UTC because KubeletNotReady (PLEG is not healthy: pleg has yet to be successful)

That means it was overloaded with too many pods. That might be the cause of

E0504 07:30:40.919989 15 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has been invalidated, token lookup failed]"

too

[balait4]

Ok I see the URL is valid one that my bad and the issue was with log-1, db-1. Last time it happened on 4th of May.

In all these worker we see "Error updating node status" & "failed to update lease, error".

[balait4]

Every time the issue happens only one server from each pair going to NotReady state (like in db workers db1 is having issue) (in logging layer log-1 is having issue.)

[balait4]

@vrutkovs We faced the same issue just 1 hour back. I'm giving the must-gather in the below link. only 3 nodes in not ready state. We are able to ssh and all working fine. I tried restarting kubelet but it didnt help. But it automatically resolve after 40 to 50 mins time. There was no resource overload in any of the node. Even master server looks good. We starting see this exactly on the same set of nodes after we upgrade to 4.12. Appreciate any help on this.
https://drive.google.com/file/d/1rfURraW3zv5kKG77wbBWQFU25R1Ul5l9/view?usp=sharing

[vrutkovs]

Is this reproducible on OKD 4.13?

[balait4]

We have same setup on dev environment where there is no issue. I'm not sure this will fix when we upgrade to 4.13 . Also the issue is intermittent and solve only by itself. restart of kubelet and kube-apiserver didnt help.
Also I saw in one post the use of http2 is the cause, is any one faced this ?

The situation described above is very similar to what we encountered. Some people said that it was because of the use of the HTTP/2.0 protocol (hereinafter referred to as h2). After searching the source code of kubelet, I found that kubelet uses the h2 protocol by default, and the specific code is implemented in the function SetTransportDefaults.

You can disable h2 by setting the environment variable DISABLE_HTTP2. After a simple verification, after explicitly setting the environment variable to disable h2, there is really no such problem in letting the connection use http1.1.

[balait4]

@vrutkovs Do you think upgrade to 4.13 is solution here? Or disabling h2 ? Concern we are getting issue only the same set of worker nodes. One node we rebuild(for someother issue) where generally this node part of this NotReady issue, after rebuild we are not getting this issue on that server. So is that some sort of certificate issue?

No network communication issue
The application is working eventhough in notready state but all pods in Terminating state which is not terminated once the kubelet able to communicate the node status with kube-apiserver then all pods terminated and started the new pod immediately.

[vrutkovs]

No idea, but if its a bug, we need it to be reproducible on 4.13 to fix it

[balait4]

Thanks