Work Queue Latency increased 10 times

[eshikhov]

When deploying Piraeus Operator Work Queue Latency increased 10 times

Deleting Custom Resources and the operator itself does not solve the problem.
I noticed that after deleting the operator, CRDs remain in the cluster.
internal.linstor.linbit.com, only after removing them Work Queue Latency returns to normal times. In the graph you can see the moment of installation and deletion of data
internal.linstor.linbit.com CRDs.

At the time of operator installation, the cluster is not empty with 130 different CRDs already deployed.

How to reproduce

Deploy Piraeus Operator configure LinstorCluster Custom Resource observe Work Queue Latency increased.
Removing Linstore CRDs (group internal.linstor.linbit.com) normalizes response times.

Environment

• OS: Talos v1.8.3
• Kubernetes version: v1.30.7
• Piraeus Operator: v2.7.1

[WanzenBug]

I guess the apiserver is not optmized for many CRDs? I wonder what it is actually doing that takes so long...

[vladimirfx]

I guess the apiserver is not optmized for many CRDs? I wonder what it is actually doing that takes so long...

20% more CRDs slow down service in order of magnitude. So I think there is something 'special' in these CRDs.

[WanzenBug]

20% more CRDs slow down service in order of magnitude

Does this happen with all CRDs? In any case, this sounds like a Kubernetes issue to me.

[eshikhov]

No, only when LINSTOR's CRDs are installed
Installing other CRDs, even more than LINSTOR's, has no negative impact

[vladimirfx]

It seems that is not a Piraeus issue but Linstore, see LINBIT/linstor-server#428