[EPIC] Resiliency in shared mode

[jp-gouin]

The kubelet and the server should be resilient to pod restart.
The kubelet has issue restarting due to the webhook cert
The server has issue restarting due to the boostrap secret

Issue encountered while using persistence.type: dynamic to persist ETCD data:

If the server pod IP changes (pod killed, reschedules, node restarted,...) the server fail to come up again:

time="2025-01-03T15:29:35Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3k-mycluster-server-0-dc22a3d8=https://10.244.3.241:2380], expect: k3k-mycluster-server-0-dc22a3d8=https://10.244.3.243:2380"

The shared mode use the embedded ETCD which use the local IP (Pod IP) to register the member. When the Pod IP changes , ETCD fails to start.

Potential solutions

Set the proper ETCD startup config using the headless service or services

etcd-arg:
  - --initial-cluster=... 
  - --advertise-client-urls=...
  - --initial-advertise-peer-urls=...

This remains challenging due to the embedded nature of ETCD. The server pod is not considered running until ETCD is running and ETCD won't start because the dns resolution will fail.

Use Sqlite in shared mode instead of ETCD

This will require some rework of the boostrap part which kubelet currently rely on to connect to the cluster.
We could use --write-kubeconfig to directly store the kubeconfig for the kubelet

[enrichman]

While investigating this issue I've seen one of the problem is related to the bootstrap secret. When the server pod is restarted the old secret are lost, and recreated.

Another issue is that deleting the bootstrap secret will not recreate it.

I'm trying to see if mounting an external secret where to store the certs is possible, and if it works. If yes this should solve most of the issues. We will need to think about a way to provide already existing, or custom certs though.

[enrichman]

Trying to watch an owned resource (Secret) I'm facing an infinite reconciliation loop.

After a bit of investigation I've seen that this is because of the Webhook certs being updated at every loop. At the moment the reconciliation is pretty convoluted, and adding the fix is not trivial, or possible, without a small refactor.

I'll open a small PR with some of them. Nice related article: https://ahmet.im/blog/controller-pitfalls/