Keda not scaling based on kafka scaler even though metric thresholds have been crossed.

[hh-sushantkumar]

This is my scaler configuration.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    meta.helm.sh/release-name: dummy-service
    meta.helm.sh/release-namespace: dummyns
    scaledobject.keda.sh/transfer-hpa-ownership: "true"
  labels:
    app.kubernetes.io/instance: dummy-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: app
    app.kubernetes.io/service: dummy-service-app
    helm.sh/chart: app-1.0.14
    scaledobject.keda.sh/name: dummy-service-app
  name: dummy-service-app
  namespace: dummyns
spec:
  cooldownPeriod: 300
  fallback:
    failureThreshold: 3
    replicas: 2
  idleReplicaCount: 1
  initialCooldownPeriod: 120
  maxReplicaCount: 10
  minReplicaCount: 2
  pollingInterval: 30
  scaleTargetRef:
    name: dummy-service-app
  triggers:
  - metadata:
      value: "80"
    metricType: Utilization
    type: cpu
  - metadata:
      desiredReplicas: "3"
      end: 0 20 * * *
      start: 0 11 * * *
      timezone: America/Los_Angeles
    type: cron
  - metadata:
      value: "80"
    metricType: Utilization
    type: memory
  - authenticationRef:
      name: kafka
    metadata:
      bootstrapServers: <REDACTED>
      consumerGroup: dummy-service
      lagThreshold: "2"
      sasl: scram_sha512
      tls: enable
      topic: dummy-topic
      unsafeSsl: "false"
    metricType: AverageValue
    type: kafka
  - metadata:
      metricName: nginx_ingress_controller_requests
      query: sum(irate(nginx_ingress_controller_requests{ingress="dummy-service-app-external",controller_namespace="kube-system",controller_class="k8s.io/external"}[1m]))
      serverAddress: http://prometheus-server.kube-system
      threshold: "10"
    type: prometheus
status:
  conditions:
  - message: ScaledObject is defined correctly and is ready for scaling
    reason: ScaledObjectReady
    status: "True"
    type: Ready
  - message: Scaling is performed because triggers are active
    reason: ScalerActive
    status: "True"
    type: Active
  - message: No fallbacks are active on this scaled object
    reason: NoFallbackFound
    status: "False"
    type: Fallback
  - status: Unknown
    type: Paused
  externalMetricNames:
  - <REDACTED>
  - <REDACTED>
  - <REDACTED>
  health:
    <REDACTED>:
      numberOfFailures: 0
      status: Happy
  hpaName: keda-hpa-dummy-service-app
  lastActiveTime: "2024-11-28T19:58:49Z"
  originalReplicaCount: 1
  resourceMetricNames:
  - cpu
  - memory
  scaleTargetGVKR:
    group: apps
    kind: Deployment
    resource: deployments
    version: v1
  scaleTargetKind: apps/v1.Deployment

Noticing this error on the keda-operator pod.

2024-11-28T18:06:19Z	ERROR	scale_handler	error getting scale decision	{"scaledObject.Namespace": "apps", "scaledObject.Name": "demo-service-app", "scaler": "kafkaScaler", "error": "scaler with id 3 not found, len = 0, cache has been probably already invalidated"}

which results in no scaling operation even though on the HPA the threshold for the scaler has been crossed.

[hh-sushantkumar]

Not sure if any one gets a similar issue, but reinstallation of the application (might be conincidence) and increase in the number of partitions helped with the testing.