Masters and workers hung after some days - High cpu load

[krzysiekkurowski]

Hi
My OKD version is 4.7.0-0.okd-2021-03-07-090821
I have 3 master nodes (6cores, 20GB ram) and 3 workers(4cores, 12GB ram).
Hosted and configured on premise, integrated with VMware vSphere 7.1
A few projects, overall the load is quite light.
However, after a few days of working, the master and workers stop responding.
Random - one master0, sometime worker2., next worker1 ..etc
I cant login to server while problem exists - all I can do is reset server via vmware (poweroff, and on).
The cpu load is so high that it is impossible to log into the machine and check what is going on.

When I checked processes during normal works - it looks quite normal - ~30% cpu load,
The most cpu consuming processes are kube-apiserver, ovn-controller, ovsdb-server, etcd, kube-controller, NetworkManager, prometheus
Has anyone had such problems? ...

Best Regards
Krzysztof

[JaimeMagiera]

Please upload a must-gather bundle.