Pod identity based authentication

[prashathsenthil]

Hi,

I have deployed the latest KEDA version 2.8.0 in my EKS Cluster (version v1.22.10-eks-84b4fe6), I'm trying to leverage KEDA for AWS SQS Queue scaler via Pod identity based authentication, however when I follow the documentation https://keda.sh/docs/2.8/scalers/aws-sqs/#example and deploy the scaledobject and triggerauth with the below manifests targeting my POD with IRSA enabled, I could see errors that indicates KEDA is trying to use the underlying worker node's IAM Role to perform sts:AssumeRole action on the IAM role annotated to my POD's ServiceAccount.

Ideally I would like KEDA operator and metric server to use my POD IAM role to connect with SQS Queue. Was looking at the scaler (https://github.com/kedacore/keda/blob/main/pkg/scalers/aws_sqs_queue_scaler.go) and aws IAM auth code (https://github.com/kedacore/keda/blob/main/pkg/scalers/aws_iam_authorization.go) not sure if my above requirement is feasible. Any help/guidance would be very useful.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-aws-credentials
  namespace: my-test-namespace
spec:
  podIdentity:
    provider: aws-eks

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: aws-sqs-queue-scaledobject
  namespace: my-test-namespace
spec:
  scaleTargetRef:
    name: my-test-scaledobject
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: keda-trigger-auth-aws-credentials
    metadata:
      queueURL: https://sqs.us-east-1.amazonaws.com/<my_aws_account>/my-test-queue
      queueLength: "2"
      awsRegion: "us-east-1"
      identityOwner: pod

My POD IRSA works as expected as I was able to test it out by pushing messages to Queue and was able to have the POD read them, also below is the POD STS getcallerIdentity O/P,

{'UserId': 'xxxxxx', 'Account': 'xxxxx', 'Arn': 'arn:aws:sts::xxxxx:assumed-role/consumer-role/botocore-session-xxxxxx', 'ResponseMetadata': {'RequestId': 'xxxxxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxx', 'content-type': 'text/xml', 'content-length': '500', 'date': 'Tue, 23 Aug 2022 02:34:24 GMT'}, 'RetryAttempts': 0}}

[vineetsharma883]

Hey, did you find any solution for this ?

[tbondarchuk]

Looks like docs are either misleading or incomplete. They kinda imply that pod identity will "just work" while obviously it does not or just requires some extra steps.

But anyway, after playing with this I believe that using dedicated keda-operator irsa role with minimal read only permissions is a better way: giving same level of access that app's pod have to completely separated controller is probably not a good idea.

Or if it's really needed then perhaps allow keda's role to assume pod's role and then it should work, theoretically.

[tomkerkhove]

Any suggestion how we can improve our docs?

[tbondarchuk]

@tomkerkhove Perhaps add clear statement that if you want to use irsa roles then best use operator's identity and irsa role and if you are using static credentials in pod's env vars or secret then pod identity is way to go? Because after reading docs it seems like using pod identity with auth set to aws-eks will allow operator to use pod's irsa role effortlessly, which in fact is not true (unless everyone os missing some non obvious config setup?)

Pod identity based authentication: podIdentity.provider - Needs to be set to either aws-kiam or aws-eks on the TriggerAuthentication and the pod/service account must be configured correctly for your pod identity provider.

I guess some example of correct configuration would be nice)

Anyway, imo using operator'a role with read only sqs permissions is much better setup then allowing it to "highjack" pod's role and getting write permissions. Got it implemented, works seamlessly, very nice! Thanks a lot for the awesome product :)

[afirth]

💯 and thanks for the discussion. very unclear that podIdentity with aws-eks would need stsAssumeRole - makes sense and I was wondering how it could work. Much more secure to give keda sqs/cloudwatch specific role I guess, and simpler.

[tomkerkhove]

@afirth @tbondarchuk I have not used AWS before so I was wondering - Does any of you want to send a PR to help fellow AWS-users?

[vincentgna]

I was assuming KEDA would work like CSI Secrets Provider -> but of course the volume mount in CSI is done under the pod identity. Whereas the KEDA operator runs separately from the pod workload(s).

at the same time:

But anyway, after playing with this I believe that using dedicated keda-operator irsa role with minimal read only permissions is a better way: giving same level of access that app's pod have to completely separated controller is probably not a good idea.

this makes a very good argument for providing the operator with limited read permissions as indeed the operator should not get the same permissions as the pod identity.

[eddy8700]

Is this issue resolved now ?

[vineetsharma883]

Restarting the keda deployments worked in my case

[eddy8700]

@vineetsharma883 restarting the keda deployment does not worked out for me