Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

replace all kubernetes certificates including ca after that could see kubectl logs and exec is not working #2015

Closed
rajibul007 opened this issue Feb 5, 2020 · 6 comments
Closed

replace all kubernetes certificates including ca after that could see kubectl logs and exec is not working #2015

rajibul007 opened this issue Feb 5, 2020 · 6 comments
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@rajibul007
Copy link

rajibul007 commented Feb 5, 2020 •
edited
Loading

kubeadm version (use kubeadm version):
1.15.4 and 1.16
Environment: : kubeadm onenode cluster barematel

  • Kubernetes version (use kubectl version): 1.15.4
  • Cloud provider or hardware configuration:kubeadm onenode cluster barematel ,ubuntu 18.04
  • OS (e.g. from /etc/os-release):ubuntu 18.04

What happened?

for a security reason I need to replace all certificate including CA ,but after that could see kubectl logs and exec is not working ,,looks like kubelet authentication issue after that I have deleted the the /var/lib/kublet/client.pem and restart the kubelet to generate again but after that only static pod is coming ,weave and coredns is not comig up

How to reproduce it (as minimally and precisely as possible)?

rm -rf /etc/kubernetes

kubeadm init phase certs all
kubeadm init phase kubeconfig all
kubeadm init phase control-plane all
kubeadm init phase etcd local
rm -rf /root/.kube
mkdir -p "$HOME"/.kube
sudo cp -rf /etc/kubernetes/admin.conf "$HOME"/.kube/config
sudo chown $(id -u):$(id -g) "$HOME"/.kube/config

--- after that could see kubectl logs and exec is not working
@rajibul007
Copy link
Author

root@ubuntu:/kubernetes/yamls# kubectl logs kube-apiserver-ubuntu -n kube-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

@rajibul007
Copy link
Author

apiserver log from docker logs:
0, AdditionalErrorMsg:
E0205 17:16:57.867904 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.870617 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.871240 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.872406 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.885597 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.886320 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0205 17:16:57.886669 1 authentication.go:89] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]

@rajibul007
Copy link
Author

kubelet logs:
root@ubuntu:/kubernetes/yamls# journalctl -u kubelet -n 50
-- Logs begin at Fri 2019-11-29 03:46:43 PST, end at Wed 2020-02-05 09:21:06 PST. --
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.316647 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.417324 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.499090 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.517911 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.619169 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.696818 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Unauthorized
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.719558 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.820192 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.894395 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Unauthorized
Feb 05 09:21:03 ubuntu kubelet[41998]: E0205 09:21:03.920557 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.020823 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.095203 41998 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Unauthorized
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.121654 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.222843 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.298693 41998 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Unauthorized
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.323233 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.424572 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.500704 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.525819 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.626244 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.702806 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Unauthorized
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.726716 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.827454 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.895917 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Unauthorized
Feb 05 09:21:04 ubuntu kubelet[41998]: E0205 09:21:04.928594 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.028953 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.096629 41998 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Unauthorized
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.129537 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.230150 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.302526 41998 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Unauthorized
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.331268 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.431717 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.502482 41998 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Unauthorized
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.532144 41998 kubelet.go:2267] node "ubuntu" not found
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.583915 41998 controller.go:135] failed to ensure node lease exists, will retry in 7s, error: Unauthorized
Feb 05 09:21:05 ubuntu kubelet[41998]: E0205 09:21:05.632658 41998 kubelet.go:2267] node "ubuntu" not found

@rajibul007
Copy link
Author

rajibul007 commented Feb 5, 2020 •
edited
Loading

from above log seem like kubelet authentication issue so I deleted /var/lib/kubelet/pki/kubelet-client-current.pem to recreate again ,,,but
after that all pod (weave ,coredns ) are not coming up

kube-system coredns-5644d7b6d9-4z5kn 0/1 Completed 0 57d
kube-system coredns-5644d7b6d9-wnjhg 0/1 Completed 0 57d
kube-system etcd-ubuntu 1/1 Running 2 57d
kube-system kube-apiserver-ubuntu 1/1 Running 2 6s
kube-system kube-controller-manager-ubuntu 1/1 Running 2 6s
kube-system kube-proxy-x9lrg 1/1 Running 1 57d
kube-system kube-scheduler-ubuntu 1/1 Running 2 6s
kube-system traefik-ingress-controller-dvffp 1/1 Running 0 57d
kube-system weave-net-d6lks 1/2 CrashLoopBackOff 2 57d

kubelet log_
Feb 05 09:26:28 ubuntu kubelet[45176]: weave-cni: unable to release IP address: Delete http://127.0.0.1:6784/ip/172d7896dad4db6bf9799d6c6e996b1f256494978c3b49564f239573f6294900: dial tcp 127.0.0.1:6784: c
Feb 05 09:26:28 ubuntu kubelet[45176]: E0205 09:26:28.708467 45176 cni.go:379] Error deleting default_nginx-deploy-green-7c67575d6c-7529z/172d7896dad4db6bf9799d6c6e996b1f256494978c3b49564f239573f6294900
Feb 05 09:26:28 ubuntu kubelet[45176]: E0205 09:26:28.709236 45176 remote_runtime.go:128] StopPodSandbox "172d7896dad4db6bf9799d6c6e996b1f256494978c3b49564f239573f6294900" from runtime service failed: r
Feb 05 09:26:28 ubuntu kubelet[45176]: E0205 09:26:28.709294 45176 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "172d7896dad4db6bf9799d6c6e996b1f256494978c3b49564f239573f6294900"}
Feb 05 09:26:28 ubuntu kubelet[45176]: E0205 09:26:28.709345 45176 kuberuntime_manager.go:658] killPodWithSyncResult failed: failed to "KillPodSandbox" for "616f47b5-0304-410f-8aa7-127959fa780b" with Ki
Feb 05 09:26:28 ubuntu kubelet[45176]: E0205 09:26:28.709370 45176 pod_workers.go:191] Error syncing pod 616f47b5-0304-410f-8aa7-127959fa780b ("nginx-deploy-green-7c67575d6c-7529z_default(616f47b5-0304-
Feb 05 09:26:33 ubuntu kubelet[45176]: W0205 09:26:33.633777 45176 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "a4468e84eb93f3c9
Feb 05 09:26:33 ubuntu kubelet[45176]: weave-cni: unable to release IP address: Delete http://127.0.0.1:6784/ip/a4468e84eb93f3c9574ceb10e7bf5eada7ef85af35952e3ad7fea14eaa7640bf: dial tcp 127.0.0.1:6784: c
Feb 05 09:26:33 ubuntu kubelet[45176]: E0205 09:26:33.786508 45176 cni.go:379] Error deleting default_nginx-deploy-main-7cc547b6f7-5dhj5/a4468e84eb93f3c9574ceb10e7bf5eada7ef85af35952e3ad7fea14eaa7640bf
Feb 05 09:26:33 ubuntu kubelet[45176]: E0205 09:26:33.787614 45176 remote_runtime.go:128] StopPodSandbox "a4468e84eb93f3c9574ceb10e7bf5eada7ef85af35952e3ad7fea14eaa7640bf" from runtime service failed: r
Feb 05 09:26:33 ubuntu kubelet[45176]: E0205 09:26:33.787700 45176 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "a4468e84eb93f3c9574ceb10e7bf5eada7ef85af35952e3ad7fea14eaa7640bf"}
Feb 05 09:26:33 ubuntu kubelet[45176]: E0205 09:26:33.787761 45176 kuberuntime_manager.go:658] killPodWithSyncResult failed: failed to "KillPodSandbox" for "3c44eb9c-4765-42f4-a50c-050bf3a06661" with Ki
Feb 05 09:26:33 ubuntu kubelet[45176]: E0205 09:26:33.787789 45176 pod_workers.go:191] Error syncing pod 3c44eb9c-4765-42f4-a50c-050bf3a06661 ("nginx-deploy-main-7cc547b6f7-5dhj5_default(3c44eb9c-4765-4
Feb 05 09:26:34 ubuntu kubelet[45176]: W0205 09:26:34.632713 45176 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "c3e04ad4cb040dcb
Feb 05 09:26:34 ubuntu kubelet[45176]: weave-cni: unable to release IP address: Delete http://127.0.0.1:6784/ip/c3e04ad4cb040dcbe2e5ee22f43276ba5e956b43461e00874659e3235d79eb67: dial tcp 127.0.0.1:6784: c
Feb 05 09:26:34 ubuntu kubelet[45176]: E0205 09:26:34.704201 45176 cni.go:379] Error deleting kube-system_coredns-5644d7b6d9-wnjhg/c3e04ad4cb040dcbe2e5ee22f43276ba5e956b43461e00874659e3235d79eb67 from n
Feb 05 09:26:34 ubuntu kubelet[45176]: E0205 09:26:34.706198 45176 remote_runtime.go:128] StopPodSandbox "c3e04ad4cb040dcbe2e5ee22f43276ba5e956b43461e00874659e3235d79eb67" from runtime service failed: r
Feb 05 09:26:34 ubuntu kubelet[45176]: E0205 09:26:34.706854 45176 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "c3e04ad4cb040dcbe2e5ee22f43276ba5e956b43461e00874659e3235d79eb67"}
Feb 05 09:26:34 ubuntu kubelet[45176]: E0205 09:26:34.707455 45176 kuberuntim

@rajibul007 rajibul007 mentioned this issue Feb 5, 2020
Closed
@neolit123
Copy link
Member

replace all kubernetes certificates including ca

kubeadm does not support CA rotation. it's a complicated process and that is why the CA is signed for 10 years. IMO an even more difficult aspect is updating the service account (sa.*) files.

we have tentative support for that in the future using the kubeadm operator:
#1698
but the timeline is unclear.

you seem to be trying to change your master IP, i'm going to have to point you at this ticket again:
#338
where the are some user ideas.

but again this is a core k8s complexity.

@neolit123
Copy link
Member

/triage support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@neolit123 @k8s-ci-robot @rajibul007