After upgrade to OKD 4.13, the kubelet status have error appear #1626
Replies: 3 comments 9 replies
-
|
Please attach (or upload to the public file sharing service) must-gather archive |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
It seem to be the bug-4521. Is this bug have appear again? |
Beta Was this translation helpful? Give feedback.
-
|
I have compare the 4.13 and 4.12, in servicemonitor have a instance call kubelet. In 4.13 kubelet service monitor is missing a line below, 4.13 missing the token tend to cannot authorize to connect the kubelet service in each node. Is it a bug or there have another method to fix it? bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am experiencing the same issue, @vrutkovs do you know if this is a bug? @JOECHONG0420 do tou have a wrk around? |
Beta Was this translation helpful? Give feedback.
-
|
@vrutkovs don't want to stress you, but this is a major issue for us. Do you have an idea of what could be wrong? or at least a workaround? |
Beta Was this translation helpful? Give feedback.
-
I have try a work around, but I feel that is very stupid. This bug will tend the user cannot see pod usage from the console. After I create the kubelet-custom servicemonitor, the pod usage appear again in the console. I don't know is this work around is good for @joyartoun you? |
Beta Was this translation helpful? Give feedback.
-
|
Finally, I got the perfect answer. @joyartoun The main problem /etc/kubernetes/kubelet-ca.crt is missing the cert. And /etc/kubernetes/kubelet-ca.crt is control by machine config. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, We have the same issue after upgrading to In the
The We believe because of this, prometheus cannot authentication to Do you have an idea where do we need to add this CA so Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Have you copy "configmap/ After you copy to initial-kube-apiserver-server-ca, you can see the new machineconfig have generated. It will update all the node there have this file "/etc/kubernetes/kubelet-ca.crt" After that, the prometheus can access the kubelet |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Yes, I have. For some reason that doesn't generate a new machineconfig, basically it seems that nothing happens. However, adding the missing CA manually to Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Same for us, no new machine config is getting generated. |
Beta Was this translation helpful? Give feedback.
-
After I upgrade OKD to 4.13.0-0.okd-2023-05-22-052007, the kubelet service have error appear in each nodes, the error is below. anyone have idea??
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─01-kubens.conf, 10-mco-default-env.conf, 10-mco-default-madv.conf
/usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
/etc/systemd/system/kubelet.service.d
└─20-logging.conf, 20-nodenet.conf
Active: active (running) since Wed 2023-05-31 08:00:08 UTC; 1 day 19h ago
Main PID: 1372 (kubelet)
Tasks: 19 (limit: 19073)
Memory: 185.8M
CPU: 2h 53min 58.007s
CGroup: /system.slice/kubelet.service
└─1372 /usr/bin/kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/control-plane,node-role.kubernetes.io/master,node.openshift.io/os_id=fedora --node-ip=192.168.1.202 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --hostname-override= --provider-id= --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --pod-infra-container-image=quay.io/openshift/okd-content@sha256:cd7949e762391ade516277da8107713ecf632c43474f2b3fe3ec6f719fd24aca --system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi --v=2
Jun 02 03:03:58 okd-master-1 kubenswrapper[1372]: E0602 03:03:58.381286 1372 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=4678241471116388661691541425963361895, SKID=, AKID=AF:20:8E:DA:76:C1:55:E6:BA:40:FF:53:A4:7B:11:33:4E:FE:C2:E8 failed: x509: certificate signed by unknown authority"
Jun 02 03:04:02 okd-master-1 kubenswrapper[1372]: E0602 03:04:02.773139 1372 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=4678241471116388661691541425963361895, SKID=, AKID=AF:20:8E:DA:76:C1:55:E6:BA:40:FF:53:A4:7B:11:33:4E:FE:C2:E8 failed: x509: certificate signed by unknown authority"
Jun 02 03:04:04 okd-master-1 kubenswrapper[1372]: E0602 03:04:04.308326 1372 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=4678241471116388661691541425963361895, SKID=, AKID=AF:20:8E:DA:76:C1:55:E6:BA:40:FF:53:A4:7B:11:33:4E:FE:C2:E8 failed: x509: certificate signed by unknown authority"
Jun 02 03:04:08 okd-master-1 kubenswrapper[1372]: E0602 03:04:08.455379 1372 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=4678241471116388661691541425963361895, SKID=, AKID=AF:20:8E:DA:76:C1:55:E6:BA:40:FF:53:A4:7B:11:33:4E:FE:C2:E8 failed: x509: certificate signed by unknown authority"
Jun 02 03:04:10 okd-master-1 kubenswrapper[1372]: I0602 03:04:10.886003 1372 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-apiserver/kube-apiserver-okd-master-1" status=Running
Jun 02 03:04:10 okd-master-1 kubenswrapper[1372]: I0602 03:04:10.886155 1372 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-controller-manager/kube-controller-manager-okd-master-1" status=Running
Jun 02 03:04:10 okd-master-1 kubenswrapper[1372]: I0602 03:04:10.886191 1372 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-scheduler/openshift-kube-scheduler-okd-master-1" status=Running
Jun 02 03:04:10 okd-master-1 kubenswrapper[1372]: I0602 03:04:10.886255 1372 kubelet_getters.go:182] "Pod status updated" pod="openshift-etcd/etcd-okd-master-1" status=Running
Jun 02 03:04:15 okd-master-1 kubenswrapper[1372]: E0602 03:04:15.096267 1372 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=4678241471116388661691541425963361895, SKID=, AKID=AF:20:8E:DA:76:C1:55:E6:BA:40:FF:53:A4:7B:11:33:4E:FE:C2:E8 failed: x509: certificate signed by unknown authority"
Beta Was this translation helpful? Give feedback.
All reactions