Workloads - Airbyte's Next Generation Job Framework. #42947
davinchia
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Summary
We are excited to announce a significant update to Airbyte’s infrastructure that will improve scalability and resilience - Workloads. This is automatically enabled with Airbyte Helm Chart
0.390.0
.Workloads is Airbyte's next generation Worker architecture. It is more scalable, reliable and maintainable than the previous Worker architecture. It performs particularly well in low-resource environments.
Workloads is an Airbyte-internal job abstraction decoupling the number of running jobs (including those in queue), from the number of jobs that can be started. Jobs stay queued until more resources are available or canceled. This allows for better back pressure and self-healing in resource constrained environments.
Careful users will notice two new Airbyte services 1) the Workload Api Server and the 2) Workload Launcher. Both services function together to enable Workloads. While we do not expect OSS users to interact with these services often, or at all, we still want to note the additional services to minimise surprises.
How This Impacts You
By separating scheduling and orchestration from data movement tasks, Workloads ensures a more scalable, resilient and flexible architecture and minimized maintenance of your Airbyte instance.
Next Steps
Nothing - this is automatically enabled for Community users as of Airbyte Helm Chart version
0.390.0
.We recommend users upgrading from existing deployments to pause existing connections before upgrading to minimise any errors.
Users interested in learning more can check out this blogpost and the public docs.
This set up has been extensively tested in Airbyte Cloud over the last 6 months and we are confident in its enhanced reliability and scalability. We expect users to not need to interact with Workloads on a day-to-day basis, or at all.
Nonetheless, we look forward to any feedback/improvement we can make!
Beta Was this translation helpful? Give feedback.
All reactions