ScaledObject with Prometheus trigger scales deployment to 0 on creation #4398

raumx · 2023-03-22T02:27:00Z

raumx
Mar 22, 2023

Hello, I've been stuck on this issue for several days, may be someone could help me.
I got a deployment with a Prometheus metric, querying with PromQL I get a valid value for the metric.
Next I create a SO with prometheus trigger and just right away my deployment scales down to 0.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sp-so-2
  namespace: sp
  labels:
    app: spanel-sp
spec:
  scaleTargetRef:
    name: spanel-2
  pollingInterval: 15
  cooldownPeriod:  300
  idleReplicaCount: 0
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: prometheus
    metadata:
      metricName: supervisor_ws_count
      serverAddress: http://prometheus-server.monitoring.svc.cluster.local
      threshold: '10'
      query: sum(supervisor_ws_count{app="spanel-2"})

There are no errors in keda operator logs:

2023-03-17T06:38:08Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "eb5ad84c-16c9-4874-aa2f-36e03da90743"}
2023-03-17T06:38:08Z    INFO    Adding Finalizer for the ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "eb5ad84c-16c9-4874-aa2f-36e03da90743"}
2023-03-17T06:38:08Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "eb5ad84c-16c9-4874-aa2f-36e03da90743", "resource": "apps/v1.Deployment", "name": "spanel-2"}
2023-03-17T06:38:08Z    INFO    Creating a new HPA      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "eb5ad84c-16c9-4874-aa2f-36e03da90743", "HPA.Namespace": "sp", "HPA.Name": "keda-hpa-sp-so-2"}
2023-03-17T06:38:08Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "eb5ad84c-16c9-4874-aa2f-36e03da90743"}
2023-03-17T06:38:08Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "ec6c943a-3179-4316-927c-1a7ad78358ab"}
2023-03-17T06:38:08Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "d187349f-5e59-4ba1-8c8b-62171489fc80"}
2023-03-17T06:38:08Z    INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject idleReplicaCount    {"scaledobject.Name": "sp-so-2", "scaledObject.Namespace": "sp", "scaleTarget.Name": "spanel-2", "Original Replicas Count": 1, "New Replicas Count": 0}
2023-03-17T06:38:23Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"sp-so-2","namespace":"sp"}, "namespace": "sp", "name": "sp-so-2", "reconcileID": "0fb6aee7-4838-4133-a8dd-3791e3cbb361"}

I'm pretty sure the metric is accessable for HPA, manually scaling to 1 pod keda can view the metric value:

spec:
  maxReplicas: 5
  metrics:
  - external:
      metric:
        name: s0-prometheus-supervisor_ws_count
        selector:
          matchLabels:
            scaledobject.keda.sh/name: sp-so-2
      target:
        averageValue: "2"
        type: AverageValue
    type: External
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: spanel-2
status:
  conditions:
  - lastTransitionTime: "2023-03-17T06:38:23Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2023-03-17T06:42:53Z"
    message: 'the HPA was able to successfully calculate a replica count from external
      metric s0-prometheus-supervisor_ws_count(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:
      sp-so-2,},MatchExpressions:[]LabelSelectorRequirement{},})'
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2023-03-17T06:43:38Z"
    message: the desired count is within the acceptable range
    reason: DesiredWithinRange
    status: "False"
    type: ScalingLimited
  currentMetrics:
  - external:
      current:
        averageValue: "1"
        value: "0"
      metric:
        name: s0-prometheus-supervisor_ws_count
        selector:
          matchLabels:
            scaledobject.keda.sh/name: sp-so-2
    type: External
  currentReplicas: 1
  desiredReplicas: 1

but out of a sudden in a couple of minutes it scales down to 0 again even the metric persists positive.
What am I doing wrong? I suppose it shoudn't get turned off while creating a SO object..

raumx · 2023-03-22T02:42:45Z

raumx
Mar 22, 2023
Author

If the current metric value is 0 while creating SO, as well it resets pods to 0 regardless cooldownPeriod or pollingInterval, just right away.
Any ideas?

0 replies

JorTurFer · 2023-04-03T22:43:02Z

JorTurFer
Apr 3, 2023
Maintainer

Hi,
Based on the logs, I think that KEDA is scaling to zero because the metric is zero. Could you check if the value is the same that you expect?

0 replies

zroubalik · 2023-04-04T10:07:44Z

zroubalik
Apr 4, 2023
Maintainer

Also, are you querying metrics from the app that is a scaleTarget as well? In this case you should not scale to 0 at all, because once you do this, you lose the source of the metric.

Remove the idleReplicaCount from your SO, it will scale between 1 and 5.

0 replies

FaraSys · 2023-04-28T14:27:05Z

FaraSys
Apr 28, 2023

@raumx Hello. I have the same problem. When i use idleReplicaCount my deployment scaled to 0 pod.
If I remove the idleReplicaCount parameter, nothing happens. I see in the logs that

Scaler for scaledObject is active	{"scaledObject.Namespace": "applications", "scaledObject.Name": "scale-myapp-web", "scaler": "prometheusScaler", "metricName": "s0-prometheus-prometheus"},

and scale does not work.
My SO:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scale-myapp-web
  namespace: applications
  labels:
    app: myapp-web
spec:
  scaleTargetRef:
    name: rpt-web
  minReplicaCount: 1
  maxReplicaCount: 5
  pollingInterval: 15
  cooldownPeriod:  120
  triggers: 
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-stack-kube-prom-prometheus.monitoring.svc.atk8s:9090
        #      metricName: prometheus-http_requests_total
      threshold: '10'
      query: sum(irate(nginx_ingress_controller_requests{service="myapp-web"}[1m])) by (service)

Any idea?

5 replies

JorTurFer May 1, 2023
Maintainer

idleReplicaCount allows the scaling to 0 https://keda.sh/docs/2.10/concepts/scaling-deployments/#idlereplicacount
If your metric is returning 0, and you have it, you are allowing the scaling to zero (similar as setting minReplicaCount:0)
Based on your query, you are scaling with the incoming traffic to the pod, so definitively you should avoid idleReplicaCount because otherwise your workload will scale to 0 if there isn't any load, and once that has happened, you won't scale from 0 never because you can't answer the incoming requests

FaraSys May 2, 2023

Thanks for the answer.
I read about idleReplicaCount. Understood the mechanism of work.
I removed this option, but why doesn't scale my workload from 1 to 3 in this case?
According to the logs, scaler is active when metric is not null.
In the next discussion, I attached all the logs.
#4484

JorTurFer May 2, 2023
Maintainer

The scaler is active when metric is > 0

FaraSys May 2, 2023

my metric > 0. Look my logs:

2023-05-02T06:39:04Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "applications", "scaledObject.Name": "scale-rpt-web", "scaler": "prometheusScaler", "metricName": "s0-prometheus-prometheus", "metrics": [{"metricName":"s0-prometheus-prometheus","metricLabels":null,"timestamp":"2023-05-02T06:39:04Z","value":"5200m"}], "activity": true, "scalerError": null}

JorTurFer May 2, 2023
Maintainer

So, if your metric is > 0, the ScaledObject is Active and it won't scale to 0. If you check the query result during the time, does it always return > 0?

FaraSys · 2023-05-02T07:01:46Z

FaraSys
May 2, 2023

When I try load my pods, the metrics are returned > 0. As soon as I turn off requests for pods, then the metrics are returned 0
At the moment my depl scaled to 1 pod. I want when HTTP requests to my pod > 10 that the scaleObject would increase the number of replicas to 2 or 3 pods. But that doesn't happen. Even when there are http requests and scaleObject is active

3 replies

JorTurFer May 2, 2023
Maintainer

Could you describe the HPA durign that period? KEDA generates an HPA named keda-hpa-YOUR_SCALEDOBJECT_NAME, there we will see the reason behind not scaling. Are you scrapping KEDA operator metrics? there you can see the returned value on each moment

FaraSys May 2, 2023

root@kmaster-01:~# kubectl describe horizontalpodautoscalers.autoscaling -n applications keda-hpa-scale-myapp-web 
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                 keda-hpa-scale-myapp-web
Namespace:                                            applications
Labels:                                               app=myapp-web
                                                      app.kubernetes.io/managed-by=keda-operator
                                                      app.kubernetes.io/name=keda-hpa-scale-myapp-web
                                                      app.kubernetes.io/part-of=scale-myapp-web
                                                      app.kubernetes.io/version=2.10.1
                                                      scaledobject.keda.sh/name=scale-myapp-web
Annotations:                                          <none>
CreationTimestamp:                                    Sat, 29 Apr 2023 15:56:35 +0300
Reference:                                            Deployment/myapp-web
Metrics:                                              ( current / target )
  "s0-prometheus-prometheus" (target average value):  <unknown> / 10
Min replicas:                                         1
Max replicas:                                         5
Deployment pods:                                      1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric applications/s0-prometheus-prometheus/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: scale-myapp-web,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-prometheus.external.metrics.k8s.io)
Events:
  Type     Reason                   Age                        From                       Message
  ----     ------                   ----                       ----                       -------
  Warning  FailedGetExternalMetric  4m54s (x15887 over 2d18h)  horizontal-pod-autoscaler  unable to get external metric applications/s0-prometheus-prometheus/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: scale-myapp-web,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-prometheus.external.metrics.k8s.io)

JorTurFer May 2, 2023
Maintainer

Based on the HPA, I guess that it doesn't work at all :(
If this discussion is yours, let's continue there

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ScaledObject with Prometheus trigger scales deployment to 0 on creation #4398

{{title}}

Replies: 5 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ScaledObject with Prometheus trigger scales deployment to 0 on creation #4398

raumx Mar 22, 2023

Replies: 5 comments · 8 replies

raumx Mar 22, 2023 Author

JorTurFer Apr 3, 2023 Maintainer

zroubalik Apr 4, 2023 Maintainer

FaraSys Apr 28, 2023

JorTurFer May 1, 2023 Maintainer

FaraSys May 2, 2023

JorTurFer May 2, 2023 Maintainer

FaraSys May 2, 2023

JorTurFer May 2, 2023 Maintainer

FaraSys May 2, 2023

JorTurFer May 2, 2023 Maintainer

FaraSys May 2, 2023

JorTurFer May 2, 2023 Maintainer

raumx
Mar 22, 2023

Replies: 5 comments 8 replies

raumx
Mar 22, 2023
Author

JorTurFer
Apr 3, 2023
Maintainer

zroubalik
Apr 4, 2023
Maintainer

FaraSys
Apr 28, 2023

JorTurFer May 1, 2023
Maintainer

JorTurFer May 2, 2023
Maintainer

JorTurFer May 2, 2023
Maintainer

FaraSys
May 2, 2023

JorTurFer May 2, 2023
Maintainer

JorTurFer May 2, 2023
Maintainer