okd 4.7 server down after etcd certificates expire

[michelsterckx]

Hi,
Our OKD 4.7 is down since a few days after the certificates used for the etcd database expired. These expired 3 years after the initial install. According to OKD documentation these should be automatically rotated which did not happen causing the cluster to go down.

etcd location: /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-peer/

I didn't find any documentation how to renew the certificates manually. Any idea what we could do to get this working again?
regards,

[JaimeMagiera]

Please see this comment...

#1630 (reply in thread)

[michelsterckx]

Thanks for the reply, Unfortunately it seems we are not in the same situation:

[root@pvx9okdmastern1 ~]# export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
[root@pvx9okdmastern1 ~]# oc get csr
The connection to the server localhost:6443 was refused - did you specify the right host or port?

So the kube-apiserver is not running because the etcd cluster is down. Any ideas?

[vrutkovs]

According to OKD documentation these should be automatically rotated

Since OKD 4.9 4.8 only

[michelsterckx]

If I look at the docs for 4.7 it does seem that's also 'system managed': https://docs.okd.io/4.7/security/certificate_types_descriptions/etcd-certificates.html

[vrutkovs]

They are system managed, as in "being created by system", but it doesn't mean they are being rotated. See issues.redhat.com/browse/OCPBUGS-7198.