Cloning/Duplicating Messages

[newlandk]

Are there any best practices to on how to duplicate a message and then perform different actions on the cloned message?

Sometimes we find it necessary to "tee" a pipeline and perform a slightly different action and send to a different output (or same output).

Current Workaround

Assuming the base document is a map/dict you can use bloblang to duplicate the base message multiple times within a bloblang mapping in a list/array. Following that, I can apply a unarchive on json_array (as a top level processor) to get multiple documents. This gets a bit messy with different documents coming from the same source and is a bit challenging since unarchive or any one-to-many has challenge within branches (not being run as a top level processor).

Suggestion/Thoughts

A clone or tee processor that has behavior similar to a branch operator except each branch results in a duplicate of the initial document with each branch allows a different array of processors to be applied to the initial document entering the processor.

I'm sure there are challenges with this suggestion, but would greatly appreciate thoughts and feedback if there are better ways to approach this challenge. Thank you!

[Jeffail]

Hey @newlandk, there's lots of different ways of doing this sort of stuff so it's hard to recommend one standard way. Normally I'd say to use a broker output with a fan_out and output level processors to do this, but then you're limited to having single threaded processing per output.

Another way is to use https://www.benthos.dev/docs/components/processors/select_parts to select a message multiple times parts: [ 0, 0 ] and then use the bloblang function batch_index to identify which index you're mapping.

A cleaner way (in my opinion) is to use branches to process different masks of a message that combines both duplicates, you could shape a message to look something like this {"id":"1","foo":{"cry":"meow"},"bar":{"cry":"woof"}} and then use request_map to mask the one you're interested in, and result_map to isolate the results.

[newlandk]

Thank you @Jeffail. I will have to take a look into those options. You insight is greatly appreciated.