Running Streams In Kubernetes Cluster

[bn-gmoss]

Hello,

I'm trying to use Benthos to solve some data translation issues in a kubernetes cluster I'm building. The goal is to have Benthos running on a pod within the cluster and to have other pods send it api queries containing data in different formats. Benthos would then take that data and convert it into a desired format and return it. Ultimately, I'd like to have Benthos handle dozens of different translations between data formats and be managed through helm, so we can use CI/CD pipelines to control and deploy new data translations automatically.

I think streams would be a perfect fit for this problem, however I'm struggling to find examples of what I'm trying to do. The documentation on the api only has options for creating the yaml through the api itself. We want to be able to use ci/cd to manage all the yaml running on Benthos in one place, rather than having each pod connecting to it need to specify what they require and manage their own Benthos yaml.

The README found here: https://github.com/benthosdev/benthos-helm-chart states that when running Benthos in streams mode in kubernetes, we need to have all streams yaml contained in a single config file, which seems to contradict the documentation found here: https://www.benthos.dev/docs/guides/streams_mode/using_config_files

when I try to adapt the sample config from the benthos-helm-chart repo to add a pipeline:

kind: ConfigMap
metadata:
  name: benthos-streams
data:
  hello.yaml: |
    input:
      generate:
        mapping: root = "hello"
        interval: 5s
        count: 0
    output:
      stdout:
        codec: lines
  aaaaa.yaml: |
    input:
      generate:
        mapping: root = "AAAAAAAAAA"
        interval: 2s
        count: 0
    output:
      stdout:
        codec: lines
  test.yaml: |
    input:
      http_server: {}
    pipeline:
      threads: 4
      processors:
      - mapping: |
          root = {
            "id": this.user.id,
            "content": this.body.content
          }
    output:
      http_server: {}

I get the following error:
Stream configuration file read error: failed to load config '[ /streams/test.yaml]': line 6: unable to infer processor type from candidates: [mapping]

If someone could point out where I've erred, or else point me to a good example of someone else doing what I'm trying to do, I'd be very grateful.

Thank you for all the work you're doing on this platform, it's really exciting stuff!

[mihaitodor]

Hey @bn-gmoss! That Helm chart needs some attention, unfortunately...

The documentation on the api only has options for creating the yaml through the api itself.

See here. There are 2 separate streams in 2 separate yaml files. Additionally, you should read this to see how the -r flag works, so you can reuse components in multiple streams.

we need to have all streams yaml contained in a single config file

Yeah, that should probably be reworded a bit. While you can volume mount as many ConfigMaps as you want here, this code will only load the yamls found in /streams, so that limits you to having to use a single ConfigMap that you specify in streams.streamsConfigMap here in which you can place multiple files as keys. I see there's an issue here to enhance this and allow users to pass in arbitrary args, so it will be addressed eventually.

I get the following error

That's because the appVersion here is out of date. You can set the tag here to the latest release (v4.10.0) and it should work. The mapping processor was introduced in v4.5.0.

Hope this is enough to get you going for now.

[ablyler]

Thanks, that was helpful!

I went ahead and created a PR to update the appVersion redpanda-data/redpanda-connect-helm-chart#37

[bn-gmoss]

Thanks for your help, this clarifies a lot and we were able to get it working!