Prometheus scaler loose metrics after scaling #5611
-
|
I have a setup with a deployment with Keda scaler with Prometheus metrics. Initially the setup works fine, metrics are received at Keda and scaling is triggered. But as soon as scaling is happened, Keda start reporting metrics collection issues, compute the replica count issues, and it does not scale back. Before scaling -> After scaling -> Configuration -> |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
I was able to find the issue. The issue was in the Prometheus query I have used for the second metric "avg_response_time_keda", it wasn't aggregated. So after scaling, query was returning multiple separate values for each replica. Keda failed to handle this and failed. The issue was solved after adding an averaging operator to the query. |
Beta Was this translation helpful? Give feedback.
I was able to find the issue. The issue was in the Prometheus query I have used for the second metric "avg_response_time_keda", it wasn't aggregated. So after scaling, query was returning multiple separate values for each replica. Keda failed to handle this and failed. The issue was solved after adding an averaging operator to the query.