Skip to content

How to run parallel pipelines #2097

Closed Answered by mihaitodor
eduardo-fp-romao asked this question in Q&A
Discussion options

You must be logged in to vote

Hey @eduardo-fp-romao 👋 I think you're confusing a few things here. The pipeline is a top level config section which lets you run the sequence of processors under pipeline.processors in parallel against batches of messages emitted by your input(s) as dictated by pipeline.threads. If you don't configure any batching, then individual messages are considered batches of size 1. You can, additionally, use a parallel processor in your pipeline.processors which can have one or more child processors under the processors field and it also has a cap field which lets you control how many individual messages from a batch it should process in parallel.

In your case, if the employee_reporter processor …

Replies: 1 comment 1 reply

Comment options

You must be logged in to vote
1 reply
@eduardo-fp-romao
Comment options

Answer selected by eduardo-fp-romao
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants