enabling OAuth2 for Azure

[hesamhamdarsi]

Hi everybody,

I am going to enable OAuth2 auth for Kafka and I am following this document:
https://strimzi.io/blog/2019/10/25/kafka-authentication-using-oauth-2.0/

the document is based on running the "keycloak" as the OIDC provider. I indeed am using Azure OIDC provider
instead.
so for the parts related to issuer and jwtEndpoint in Kafka clsuter, I am using

validIssuerUri: https://login.microsoftonline.com/<My-Tenant-ID>/v2.0
jwksEndpointUri: https://login.microsoftonline.com/<My-Tenant-ID>/discovery/keys

And as the certificate for Azure is valid, I have omitted the following part from the configuration:

# tlsTrustedCertificates:
#  - secretName: ca-truststore
#    certificate: ca.crt

The kafka cluster is deployed without any issue. but when I am trying to connect to my brokers as well as request for token from the example producer application, I am receiving some errors like (related to the trust store):

ERROR java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext) (org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule)
java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
	at java.base/javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:263)
	at java.base/javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:270)
	at java.base/sun.net.www.protocol.https.HttpsClient.createSocket(HttpsClient.java:413)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:162)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
...
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
...
Caused by: java.security.KeyStoreException: problem accessing trust store at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73)
...
Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
...
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available at java.base/javax.crypto.Mac.getInstance(Mac.java:191)
...
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: An internal error occurred while retrieving token from callback handler
...
Caused by: javax.security.auth.login.LoginException: An internal error occurred while retrieving token from callback handler
...

The producer app:

apiVersion: v1
kind: Pod
metadata:
  name: kafka-client-shell
  namespace: test
spec:
  containers:
  - name: kafka-client-shell
    image: strimzi/kafka:0.20.1-kafka-2.5.0
    command: ["/bin/bash"]
    args: [ "-c", 'for((i=0;;i+=1)); do echo "Up time: \$i min" && sleep 60; done' ]
    env:
    - name: CLASSPATH
      value: /opt/kafka/libs/kafka-oauth-client-*:/opt/kafka/libs/kafka-oauth-common-*
    - name: OAUTH_TOKEN_ENDPOINT_URI
      value: https://login.microsoftonline.com/<My-tenant-ID>/oauth2/token
    volumeMounts:
    - name: truststore
      mountPath: "/opt/kafka/certificates"
      readOnly: true
  volumes:
  - name: truststore
    secret:
      secretName: kafka-client-truststore

the configurations that are applied to producer after running:

export OAUTH_CLIENT_ID=<My-Azure-AppRegistration-ID>
export OAUTH_CLIENT_SECRET=<My-Azure-AppRegistration-Secret>
export PASSWORD=truststorepassword
export KAFKA_OPTS=" \
  -Djavax.net.ssl.trustStore=/opt/kafka/certificates/kafka-client-truststore.p12 \
  -Djavax.net.ssl.trustStorePassword=$PASSWORD \
  -Djavax.net.ssl.trustStoreType=PKCS12"
  
bin/kafka-console-producer.sh --broker-list \
  my-bootstrap-svc:9093 --topic my-topic \
  --producer-property 'security.protocol=SASL_SSL' \
  --producer-property 'sasl.mechanism=OAUTHBEARER' \
  --producer-property 'sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;' \
  --producer-property 'sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler'

worth mentioning, here is the step for creating kafka-client-truststore secret that is mounted in the producer example app:

kubectl get secret/kafka-oauth-ca-cert -n test -o yaml | grep ca.crt | awk '{print $2}' | base64 --decode > kafka.crt
export PASSWORD=truststorepassword
keytool -keystore kafka-client-truststore.p12 -storetype PKCS12 -alias kafka -storepass $PASSWORD -keypass $PASSWORD -import -file kafka.crt -noprompt
kubectl create secret generic kafka-client-truststore -n test --from-file=./kafka-client-truststore.p12

I have tested different image versions in case there is issue with java runtime, but non of them working.
is there any idea?

[scholzj]

I don't think it is not really clear what is your configuration and where you get the errors. You should probably share the full configurations and full logs. From sharing some snippets it is not really clear what errors and where you get. Apart from the client image, it is also not clear what versions you use (+ strimzi/kafka:0.20.1-kafka-2.5.0 is ages old - you should probably use something newer).

[PatrickWolleb]

@hesamhamdarsi have you found a solution to get it to work? I am trying the same but I get another issue:

Strimzi config:

kind: Kafka
metadata:
  name: srf-strimzi
spec:
  kafka:
    replicas: 2
    version: 3.7.1
    config:
      log.cleanup.policy: delete,compact
      default.replication.factor: 1
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      offsets.topic.min.isr: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.roll.hours: 1
      num.network.threads: 6
      log.cleaner.delete.retention.ms: 864000000
    authorization:
      superUsers:
        - super-user
      type: simple
    resources:
      requests:
        memory: 4Gi
        cpu: 500m
      limits:
        memory: 8Gi
        cpu: 2000m
    listeners:
      - name: external
        authentication:
          type: oauth
          clientId: xxxxxxxxxxxxxxx
          clientSecret:
            key: val
            secretName: broker-oauth-secret
          validIssuerUri: https://login.microsoftonline.com/xxxxxxxxxxxxxxxxxxxx/v2.0
          jwksEndpointUri: https://login.microsoftonline.com/common/discovery/v2.0/keys
        port: 9095
        type: loadbalancer
        tls: false
        configuration:
          class: service.k8s.aws/nlb
          bootstrap:
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-central-1:xxxx:certificate/b3ca30c9-059d-4228-bbe8-a092de9d4bae"
              service.beta.kubernetes.io/aws-load-balancer-backend-ptocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9095"
              external-dns.alpha.kubernetes.io/hostname: kafka.example.com
          brokers:
          - broker: 0
            advertisedHost: kafka-0.example.com
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-central-1:xxxxxxxxxx:certificate/b3ca30c9-059d-4228-bbe8-a092de9d4bae"
              service.beta.kubernetes.io/aws-load-balancer-backend-ptocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9095"
              external-dns.alpha.kubernetes.io/hostname: kafka-0.example.com
          - broker: 1
            advertisedHost: kafka-1.example.com
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-central-1:xxxxxxxxxxxxx:certificate/b3ca30c9-059d-4228-bbe8-a092de9d4bae"
              service.beta.kubernetes.io/aws-load-balancer-backend-ptocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9095"
              external-dns.alpha.kubernetes.io/hostname: kafka-1.example.com
  zookeeper:
    replicas: 2

Error log from broker container:

ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [main]
java.lang.IllegalArgumentException: Value not specified for key ';' in JAAS config
	at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:116)
	at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)
	at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:93)
	at org.apache.kafka.common.security.JaasContext.loadServerContext(JaasContext.java:74)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:143)
	at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
	at kafka.network.Processor.<init>(SocketServer.scala:973)
	at kafka.network.Acceptor.newProcessor(SocketServer.scala:879)
	at kafka.network.Acceptor.$anonfun$addProcessors$1(SocketServer.scala:849)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
	at kafka.network.Acceptor.addProcessors(SocketServer.scala:848)
	at kafka.network.DataPlaneAcceptor.configure(SocketServer.scala:523)
	at kafka.network.SocketServer.createDataPlaneAcceptorAndProcessors(SocketServer.scala:251)
	at kafka.network.SocketServer.$anonfun$new$31(SocketServer.scala:175)
	at kafka.network.SocketServer.$anonfun$new$31$adapted(SocketServer.scala:175)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
	at kafka.network.SocketServer.<init>(SocketServer.scala:175)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:380)
	at kafka.Kafka$.main(Kafka.scala:112)
	at kafka.Kafka.main(Kafka.scala)