[source-mysql] Concurrency configuration #49849
Unanswered
alonso-cobre
asked this question in
Platform General Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Im facing some issues for having multiple sync jobs, Im expecting to have 500 sync/jobs to be running every 5 minutes, right now I have 100 ish sync/jobs running fine, but once I started adding more connections (Every connection is scheduled as a cron to run every 5 minutes), jobs runs for longer (normally they take about 1-2 minute, I have Incremental Append from AuroraMySQL to S3), sometimes they remain in a running status for hours without finishing properly, I need to cancel the sync and the next run normally finishes normally, any configuration parameters that I could look into? I am already using MAX_*_WORKERS and WORKLOAD_LAUNCHER_PARALLELISM with values ranging from 100-200, I have 3 workers (3 Kubernetes pods), I also read that TEMPORAL_WORKER_PORTS somewhat affect concurrency, I want to know how you can configure Airbyte to support this type of concurrency?
Thanks.
Beta Was this translation helpful? Give feedback.
All reactions