Scaling jobs with KEDA - Having trouble with aadpodidbinding

[sai25590]

I have a simple container that's reads a message off a service bus with ReceiveAndDelete, sleeps for 60 seconds and exists. I want to scale this as a job using scaled object in KEDA. I am following this instrcutions:
https://keda.sh/docs/1.4/concepts/scaling-jobs/
The container also has managed identity (via aadpodidbinding).
This is how my yaml looks like:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-autoscale-scaledobject
spec:
  scaleType: job
  jobTargetRef:
    parallelism: 1 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
      spec:
        metadata:
          labels:
            app: autofab-autoscale
            aadpodidbinding: azureresourceaccess
        containers:
          - name: autofab-autoscale
            image: ${acrlogin}/${autofabautoscaleimage}:${version}
            resources:
              requests:
                cpu: 1000m
            volumeMounts:
              - mountPath: /mnt/azure
                name: volume
            env:
              - name: SERVICEBUS_NAMESPACE
                value: ${sbnamespace}
              - name: AUTOSCALE_QUEUE_NAME
                value: "autofab-autoscaleq"
        restartPolicy: Never
        volumes:
          - name: volume
            persistentVolumeClaim:
              claimName: azurefile
    selector:
      matchLabels:
        app: autofab-autoscale
      
  pollingInterval: 30  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      namespace: ${sbnamespace}
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth

The problem is no jobs are created when the service bus has messages in the queue:

root@autofab-test-hopbox-vm:/home/sasubra# kubectl get jobs

NAME                            COMPLETIONS   DURATION   AGE
deployment-restart-1598954400   1/1           11s        8d ===> UNRELATED

Question 1: I have not specified any job kind anywhere apart from inside the scaled object yaml Is this right?
Question 2: Where to specify the Aadpodidbinding. Its' usually in the metadata for deployments (assume same for job) which is in top level and not at the spec/template level.

Any help is appeciated.

[sai25590]

Update: I think I found the issues. Two things:

1. We need to add metadata with aadpodidbinding at two places. One at scaledobject and another at job spec.
2. Remove selector clause from job.

Here is my new YAML:

---
# Example followed: https://github.com/ross-p-smith/Mercury/blob/2eb97992007b104581cf1381eed11d93eb950bd8/charts/kedaV1/templates/test-scaledobject.yaml
# https://keda.sh/docs/1.4/concepts/scaling-jobs/
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-autoscale-scaledobject
  labels:
    app: autofab-autoscale-scaledobject
    aadpodidbinding: azureresourceaccess
spec:
  scaleType: job
  jobTargetRef:
    parallelism: 1 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
      spec:
        metadata:
          labels:
            app: autofab-autoscale-scaledobject
            aadpodidbinding: azureresourceaccess
        containers:
          - name: autofab-autoscale
            image: ${acrlogin}/${autofabautoscaleimage}:${version}
            resources:
              requests:
                cpu: 1000m
            volumeMounts:
              - mountPath: /mnt/azure
                name: volume
            env:
              - name: SERVICEBUS_NAMESPACE
                value: ${sbnamespace}
              - name: AUTOSCALE_QUEUE_NAME
                value: "autofab-autoscaleq"
        restartPolicy: Never
        volumes:
          - name: volume
            persistentVolumeClaim:
              claimName: azurefile
      
  pollingInterval: 30  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      namespace: ${sbnamespace}
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth
---

The new problem is the container's authentication vai pod identity is failing :(

[sai25590]

One of the problem I see is that the aadpodidbinding label is not applied in the pod. Not sure why:

root@autofab-test-hopbox-vm:/home/sasubra# kubectl describe pod autofab-autoscale-scaledobject-mtxfp-c8f5v
Name:           autofab-autoscale-scaledobject-mtxfp-c8f5v
Namespace:      default
Priority:       0
Node:           aks-agentpool-36151383-vmss000015/100.73.14.140
Start Time:     Wed, 09 Sep 2020 23:07:03 +0000
Labels:         controller-uid=c3d86dff-50c3-40ec-b5a3-3ac5b0c55262
                job-name=autofab-autoscale-scaledobject-mtxfp
                scaledobject=autofab-autoscale-scaledobject

[sai25590]

Finally found the answer. The AAD pod idenity metadata should be spec.template level. Here is the working yaml:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-autoscale-scaledobject
spec:
  scaleType: job
  jobTargetRef:
    parallelism: 15 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    # We ideally want infinite but doesn't look like kubernetes supports it:
    # https://stackoverflow.com/questions/49568337/kubernetes-processing-an-unlimited-number-of-work-items
    completions: 15 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      metadata:
        name: autofab-autoscale
        labels:
          aadpodidbinding: azureresourceaccess
          app: autofab-autoscale
      # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
      name: autofab-autoscale
      spec:
        containers:
          - name: autofab-autoscale
            image: autofabtest.azurecr.io/autofab-autoscale-sasubra:v0.95
            resources:
              requests:
                cpu: 0.5m
            volumeMounts:
              - mountPath: /mnt/azure
                name: volume
            env:
              - name: SERVICEBUS_NAMESPACE
                value: autofab-test-testdc3p01-bus
              - name: AUTOSCALE_QUEUE_NAME
                value: "autofab-autoscaleq"
        restartPolicy: Never
        volumes:
          - name: volume
            persistentVolumeClaim:
              claimName: azurefile

  pollingInterval: 30  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      namespace: autofab-test-testdc3p01-bus
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth

[tomkerkhove]

Woops sorry I've missed this one. Is there anything we can improve in the docs?

We always acceps PRs on https://github.com/kedacore/keda-docs

[sai25590]

Thanks @tomkerkhove . I think the main problem was I couldn't figure out where the aadpodidbinding goes in a job object. So I guess the documentation for that should actually come from aadpodidbinding from Azure. I shall think of how this can be referenced in KEDA's docs and might send a PR out.
Meanwhile I am having other problems where I think I didn't fully understand the KEDA documentation:
#1149

[tomkerkhove]

Thanks for the PR!