Message consumer group based split scaling

[ovidiubuligan]

We have a situation where I want to split the message consumer deployment of pods and scale them independently , based on a the number of queues. In my case the queues are dictated by a few hundred customers groups Ex : myqueue/{customerGroupNr} .

Currently I can do this with keda by manually creating the few hundred deployments representing our customer groups , then defining a scaled object for each customer group .Each consumer group will have it's own number of consumers and unused consumer groups for a longer time should scale to 0.

For automating this do you think a new Operator that watches the consumer group queues and creates deployment and scaled object for each consumer group ? Or can this be done with a keda plugin ?

[zroubalik]

Hi, I think that a simple custom operator is the right fit for this usecase, as you can implement it specifically according to your needs.

Operator SDK is a simple and convenient way how to implement one: https://sdk.operatorframework.io/docs/building-operators/golang/