Support for AWS_MSK_IAM

[mbartusiak]

Hello

AWS announced support for a new SASL mechanism called AWS_MSK_IAM which enables to handle both authentication and authorisation with AWS IAM. Is there a chance to add support for that mechanism to librdkafka?

Cheers
Michal

[edenhill]

Is that its own mechanism or on top of OAUTHBEARER?

[rolandyoung]

Given that this is for MSK (Amazon Managed Streaming For Apache Kafka) and that librdkafka doesn't have direct support for Apache Streams, is there much of a use case for supporting it?

[mbartusiak]

AWS MSK is just a managed kafka cluster, utilising streams in apps is optional. You can still use basic producers and consumers

[sayantacC]

I am with AWS and a contributor to aws-msk-iam-auth repo , the jvm based plugin for integrating AWS IAM authentication and authorization as a SASL mechanism with Kafka. We would love to engage with you about adding support for the AWS IAM SASL mechanism to librdkafka.

Based on initial exploration, we have identified two possible approaches:

1. Write a librdkafka plugin to implement and register an additional sasl provider. However, as far as I could tell, librdkafka's plugin architecture will have to be extended to allow a plugin to implement and set the sasl provider based on the configured mechanism.

2. Implement AWS_MSK_IAM as another sasl mechanism within librdkafka.

We would love to learn if there are other possible approaches and your thoughts on which one to take.
Also, any suggestions on next steps would be great.

[twmb]

For potential cross-language implementation reference, I've added support for AWS_MSK_IAM in my franz-go client. I've tested it against a scrappy cluster setup, but I'd be happy to have broader testing (and I'm planning to turn down my cluster imminently).

I think that implementing the bare minimum of this sasl (similar to how I did it) would not require too much C code.

[sayantacC]

I created an issue for this feature request.