Document how to use deployment along with Kafka

[desmarchris]

Currently, you must add a new section to the values file as well as edit a config map to enable Kafka. This is shown here: https://github.com/RasaHQ/rasa-x-helm/pull/163/files

Definition of Done
As a user I can view documentation on how to enable Kafka using helm values.

[b-quachtran]

@RasaHQ/infrastructure-squad Are there any plans for supporting this out of the box in the near future?

[indam23]

You can add additional endpoints to rasa.additionalEndpoints, so it's not necessary to modify the config map. I just realized this after opening the PR above 😅

[tczekajlo]

@RasaHQ/infrastructure-squad Are there any plans for supporting this out of the box in the near future?

Not really, I mean you can already do this with the rasa helm chart as @melindaloubser1 mentioned.

For the rasa-x-helm chart, we won't add it, the goal that we have is to separate rasa-x from rasa oss.

[indam23]

I'd suggest we close this since it's already supported/possible at least.

[desmarchris]

More of my customers use Kafka than Rabbit, so I think it should be a first class citizen. We shouldn't have to send them examples on how to enable Kafka

[indam23]

That's a fair point, but RabbitMQ is a service started by the helm deployment itself, while Kafka is always external. I think (@tczekajlo correct me if I'm wrong) that is why rabbit got its own section, while Kafka endpoints have to be added in a subsection.

[desmarchris]

while Kafka is always external

are you sure? I didn't think this was the case

[indam23]

well now I'm not sure anymore 😅 But I guess since it requires forking the helm charts to make kafka a service as part of the deployment, I'd never seen anyone do it that way.

[desmarchris]

I still think there'd be value in adding a simple kafka section like we have in OSS endpoints instead of adding them to a generic area

[indam23]

In that case we can re-open the closed PR since that's what it did - @tczekajlo since it's not really a functional change and it seems it would be more intuitive for users, is that fine?

[tczekajlo]

Here are my thoughts 😄 :

1. For now, a way to add extra configuration is disabling rabbitmq (rabbitmq.enabled=false) and add Kafka section to rasa.additionalEndpoints

#values.yaml
rasa:
  additionalEndpoints:
    event_broker:
      type: "kafka"
      # security protocol
      security_protocol: SASL_PLAINTEXT
      # kafka topic
      topic: my_topic
      # kafka host
      url: localhost
      # SASL credentials
      sasl_username: username
      sasl_password: password

rabbitmq:
  enabled: false

2. @melindaloubser1 is right, rabbitmq got his own section because it's a component included in the helm chart.
3. Following the helm good practices, adding kafka section will imply that a Kafka helm chart is included and used as a sub-chart, and you can configure it through the kafka section, which is not true.

@desmarchris As there is a way to configure Kafka via values without editing the helm chart itself, my proposition would be just to document how to enable Kafka, WDYT?

cc @RasaHQ/infrastructure-squad

[desmarchris]

ok fine ;) can we add it to the documentation?

[tmbo]

I'll update the issue according to the discussion:

• we won't add native support for kafka
• we will add documentation to document how to enable kafka

[andreoid]

Open issue has been moved to JIRA: https://rasahq.atlassian.net/browse/INFRA-7