KafkaScaler - How to get rid of / How to handle "invalid offset found for topic" warnings after Topics have been recreated in Kafka for DEV ?

[QuintinBecker]

We are using KEDA in OKD as central deployment and use it for DEV and PROD branches at the same time.
Same is true for our Kafka which is used for all branches.
As we build new pipelines and reset Kafka-Topics during this proces we need to reset the stored metrics in Keda as well for this topics.

Is there a planed procedure to handling the reset of a topic that is used by scaledObjects ?

In addition we see aome of the deployments not scaling to zero in this state and assume this is linked to this problem.

Environment:
OKD 4.5
Keda 2.0
Rollout is handled by Argo v1.7.6+b04c25e

[zroubalik]

we need to reset the stored metrics in Keda as well for this topics

What do you mean by this exactly?

[QuintinBecker]

Sorry for my late response, we made some tests and chenged some points on the pipeline.
Thank you for all the info on the scaling process.

In our case we destruct the pipeline (scaled objects and deployments) and all topics linkt to it and redeploy everything. In this state the offset is default -1001 or so and there should be no scaling of the deployments till we get some events running through the pipeline.

As we have 5 partitions we get assignment of a single consumer to all of them:

[PYTHON][INFO][2020-10-06 07:11:58,962]  Error Message in Kafka: TOPIC_1 1 0
[PYTHON][INFO][2020-10-06 07:11:58,962]  % TOPIC_1 [1] reached end at offset 0

[PYTHON][INFO][2020-10-06 07:11:58,962]  Error Message in Kafka: TOPIC_1 2 0
[PYTHON][INFO][2020-10-06 07:11:58,962]  % TOPIC_1 [2] reached end at offset 0

[PYTHON][INFO][2020-10-06 07:11:58,962]  Error Message in Kafka: TOPIC_1 0 0
[PYTHON][INFO][2020-10-06 07:11:58,962]  % TOPIC_1 [0] reached end at offset 0

[PYTHON][INFO][2020-10-06 07:11:59,462]  Error Message in Kafka: TOPIC_1 4 0
[PYTHON][INFO][2020-10-06 07:11:59,462]  % TOPIC_1 [4] reached end at offset 0

[PYTHON][INFO][2020-10-06 07:11:59,464]  Error Message in Kafka: TOPIC_1 3 0
[PYTHON][INFO][2020-10-06 07:11:59,464]  % TOPIC_1 [3] reached end at offset 0

After that the Deployment is waiting and is not scaled down to 0.

Do we have to initialize an offset to all partitions to get rid of this behaviour ?

[grassiale]

HI @QuintinBecker your assumption is correct. Until all partitions won't have a committed offset, keda will not scale to 0.
I think this behaviour can be changed, but I still haven't put my mind into it.

[QuintinBecker]

Hi @grassiale thanks for the feedback.
It is not a big deal to get all partitions "initialized" on our side now that we know it is not an issue per se.
Do you plan to adopt the behavior or do you prefer a workaround on the customer side (catch offsets of -1001 and set offset to 0 by client) ?
Can you maybe have a look on my request #1190 as well ? This would make life much easier for us.

[iferca]

@QuintinBecker I'm having similar "issue", just deploying my ScaledJob I start getting this message:

invalid offset found for topic topic-test in group cg-test and partition 0, probably no offset is committed yet

and I don't see any scaling happening even if I post something in the topic while I can still consume from the terminal using kafka cli.

In your previous post you said "It is not a big deal to get all partitions "initialized" on our side" abd I undertand this solved your issue. Can you please enlight me about how did you initialized the partions, what do you mean by that?

[QuintinBecker]

Hi @iferca,
we had an issue after we had removed the topics (and all events) from Kafka and redeployed the same named topic. This is done for testing our event-driven pipeline.
We got this message because we had removed all events and there was no offset set after the topic had been created. This situation led to the deployment not scaling down to zero. So we had the deployment running even if ther has been no lag on the topic and consumergroup. In addition we are running the consumer with setting autocommit = false and have to handle each event seperately.
"Initialization" in this case means simply to create at least one Event in each patition of the topic and let the consumer set the offset correct.

If I get you right your deployments are not scaling properly even if there is lag for the consumergroup and topic.
Common Issues we have seen in the past were a mismatch of the conumsergroups in the deployment and its scaled object.
In addition keep in mind that an error within the deployment can lead to this behaviour as well. So please check the deployment itself and try to deploy it by hand.