Process SQS message acknowledgments in background #24370
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Use Cases
Currently when aws_s3 source has acks enabled it'll:
- process n messages based on
client_concurrency - wait for them all to be flushed in the sink
- process the next batch
This isn't optimal and causes a large performance degradation compared to when acks are disabled
Attempted Solutions
No response
Proposal
Similar to how I think kafka and aws_sqs sources handles the acks, it should keep reading messages and ack them in a background task (finalizer), allowing a consistent flow of messages.
References
No response
Version
0.51.1