Deprecation of Docker Compose

[nataliekwong]

Summary

Today, we are announcing that Docker Compose will be deprecated from future versions of Airbyte OSS after version 1.0 is released (expected in early August 2024).

Support for Docker Compose and running Airbyte via the run_ab_platform.sh script will end in August 2024.

What we saw

• Our users frequently experience issues when upgrading on Docker Compose, as it is very brittle to upgrade and can often result in Airbyte not restarting correctly.
• The path from a local deployment to a production is painful and requires manual mapping work. Our users have difficulty understanding the mapping of the Helm chart to Docker Compose environment variables, or config.

As Airbyte matures, we need to align on the best path forward for our users and set you up for success with Airbyte. Similar open-source companies provide Docker Compose as a deployment method for local testing. When it comes to production-readiness, Kubernetes has become the de-facto standard in the industry. Earlier this year, we also shared a survey to understand how we could improve the maintenance and upgrade experience for our own community. From your responses, we found:

• The majority of respondents deploy to production using Kubernetes
• 71% of the respondents were already familiar with Kubernetes, with a significant majority experienced in Helm.

With this data in hand, our team prioritized building a new local deployment tool, abctl, that was designed to deploy Airbyte locally using Kubernetes with minimal onboarding friction. abctl allows users to test what a production deployment would look like without needing to provide your own k8s cluster.

Next Steps

Our overall goal is to build a best-in-class tool to improve the initial user experience and ensure users are set up for a production-ready instance of Airbyte from the start, and require minimal migration.

Since our launch of abctl two months ago, we’ve observed the following signals from the adoption so far:

• 1.5x success rate of installation
• 2x more users can successfully start syncing data
• 1.5x sync speed improvement

Moving forward, as we move to deprecate Docker Compose, we want to make sure Docker Compose users can adequately migrate with enough notice and time to use abctl. To this end, we’ve created a migration path that we are beginning to test with external users. We are asking for volunteers from the Community to help ensure our migration paths can adequately cover the ways in which users currently use Docker Compose. This beta group of testers will work directly with our Engineering team to migrate existing Airbyte deployments to abctl.

To sign up for the beta group, please fill out the survey HERE.

We look forward to hearing from you!

[marcraminv]

This means all of these options will be deprecated, right?

[phammer]

Same question from my side.

[bgroff]

For ECS, we do not currently support it and do not have and plans on the roadmap to support ECS. For EC2, we have documentation here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl that explains how to use abctl on an EC2 machine.

[jadetr]

We are hosting airbyte on a ubuntu server with portainer. Traefik is acting as a reverse proxy (handling ssh, authentication.) Traefik is configured right into the docker-compose file. Any hints on the recommended upgrade path for such a setup?

[marcosmarxm]

Hello @jadetr, thanks for sharing your setup. The team is on a migration tool. I'll share with them to see if it's possible to use the current migration tool.

[bgroff]

You can read the documentation on the migration path here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl

Currently looking into how Traefik will work as an Ingress for abctl.

[ygrekconsutling]

Hi, just tried it on a fresh install on debian and got this error:

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /home/foo/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 27.0.3
 SUCCESS  Port 8000 appears to be available
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Cluster 'airbyte-abctl' created
  INFO    Namespace 'airbyte-abctl' created
  INFO    Persistent volume 'airbyte-minio-pv' created
  INFO    Persistent volume 'airbyte-volume-db' created
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created
  ERROR   Unable to fetch airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not fetch chart airbyte/airbyte: Chart.yaml file is missing

since i'm using the new install format, what's wrong and how to debug it if needed ?

[ygrekconsutling]

Finally installed the version 0.5and it worked

[whanata]

I'm currently hosting Airbyte using Docker Compose at this moment also and changes we need to make was around running Open Telemetry Collector in Docker Containers also etc.

Would be interested to make sure that using the ABCTL tool, we can still get Open Telemetry to function properly, or a pathway to use Open Telemetry to get some of the metrics from Airbyte.

We currently don't have much experience using Kubernetes, so we plan to run this using ABCTL at least for the time being in an EC2.

[bgroff]

There is documentation here for how to install Airbyte using abctl on an EC2 machine here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl

[aviad-deri]

Hi,
we are also using OSS with docker compose.
we want to help with the migration to abctl testing.
please add us to this and tell us how to migrate.
:)

[bgroff]

There is documentation on the migration here: https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#migrating-from-docker-compose-optional

[honggyu-rr]

Can we get more documentation on the migration process and what it's actually doing? E.g. will it somehow integrate my .env variables as part of the process?

[bgroff]

No, you will need to migrate from the .env variables to the values described here: https://docs.airbyte.com/deploying-airbyte/integrations/storage

[honggyu-rr]

@bgroff the guide is pretty unclear to me, someone who doesn't have kubernetes experience. I just need to be able to set my RDS instance as my external database and JOB_MAIN_CONTAINER_MEMORY_LIMIT. Any guidance?

[honggyu-rr]

@bgroff any updates? Would be great to have more information on the migration guide from docker compose, especially for us folks without kubernetes experience, as currently, the guide seems a bit sparse and too concise.

[anzelpwj-rev]

Signed up for the beta test. I've been migrating our Airbyte setup to Terraform - will there be any changes to how Terraform provisions Airbyte, or will that be unaffected?

[bgroff]

No, the Terraform configuration will be unaffected by the transition to abctl.

[magralo]

Hello all! One question i am migrating using google cloud VM…

i assume the issue that i have after running
abctl local install --migrate
is because of docker user permissions

Does anyone know how to solve this issue?

update:

This appears now when i try to check the UI:

This is why i assume the issue is starting the containers (so docker related issue)

[bgroff]

Do you have kubctl on the vm by chance? If not can you install it following the directions here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

Then can you run: kubectl get pods -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig and post the results?

[magralo]

Here you go

[magralo]

All source-related containers fail to start!

[magralo]

These are the logs of the source-related pods

E0718 05:07:35.069645 142911 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0718 05:07:35.070183 142911 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0718 05:07:35.071870 142911 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0718 05:07:35.073953 142911 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

[whanata]

Sorry might miss this in the docs, but just wondering, if we want to expose airbyte that was running through ABCTL to the outside world (change hostname from localhost to 0.0.0.0), how would we do that, could not find it in the docs and going through ABCTL, it looks like localhost is hard coded.

[alrikvidal]

Hello, need the feature too :)

[whanata]

@alrikvidal I didn't end up doing this and just running Airbyte Helm Charts directly using Kind (so it uses the Docker Containers).

I got it working (and using kubectl port forward to make sure it can be set to 0.0.0.0).

[ygrekconsutling]

Had the same issue, simply used a reverse proxy and that's it since my server already have one and already using it for other tools, very useful for the dockers instances and reconfigured tools easy to run

[bgroff]

We now have a --host flag that has been added to the abctl local install. So you can now run abctl local install --host airbyte.company.example if you want and that will set the host correctly for the ingress.

[vivekin]

Hi @nataliekwong, @bgroff

We are using community OSS version with docker compose deployed on EC2 instance.
Additionally we have configured Datadog integration, external database etc.

How can we migrate this to the new abctl command with all the modifications done to docker compose?

Already went through these resources [1] [2] [3]

But did not get the clarity whether abctl supports the above mentioned additional configurations and how to do it.
Also what exactly does the 'abctl local install --migrate' command do?

[dave-tomkinson]

We possibly do something similar; we run a script in the EC2 user_data which downloads and installs all the packages, pulls Airbyte and the docker-compose then tweaks the .env to set variable and point Airbyte at our RDS instead of the local docker pg instance.

We've done this so we don't need to care about the EC2 instance we can spin it up and down on demand and it just hooks back to RDS and carries on.

So looking to switch over to abctl but like above not necessarily 'migrate' anything... just have it point to our RDS.
Currently stuck on 0.50.4 and hoping that we'll still be able to switch to abctl.

[bgroff]

Sorry for the late reply, there are docs for that use case here: https://docs.airbyte.com/deploying-airbyte/integrations/database

[evoxmusic]

Hi, this is Romaric from Qovery. I got notified of this post, and I'm working to provide a quick install guide for Airbyte on EKS. If you are interested, please let me know.

[vivekin]

Yes, ...that would be of great help!

[evoxmusic]

Here is a very simple way to deploy Airbyte in production

https://github.com/evoxmusic/qovery-airbyte

[rtol5]

Hi, migrating with abctl was a breeze!

I was running abctl local install to upgrade to the latest version (but first I upgraded abctl itself to latest), and the installation is getting stuck with upgrading ingress-nginx from 4.11 to 4.11.1

After waiting for 30 minutes, I canceled the command and restarted it with the --verbose flag. I can see that the installation is stuck in the step below:
DEBUG helm - DEBUG: Deployment is not ready: ingress-nginx/ingress-nginx-controller. 0 out of 1 expected pods are ready

Any thoughts on how we can remediate this?

[babaMar]

Hi,

I'm running docker in rootless mode, so when running abctl local install --migrate I get an error as the deamon is not listening on unix:///var/run/docker.sock but rather on /run/user/1000/docker.sock. Is there a way to tell abctl to ping the rootless client?

[colesnodgrass]

The code currently makes some assumptions about where the docker socket is. It would make more sense to fetch this from the context itself. I'm going to add support for this to the abctl tool.

[colesnodgrass]

As the docker cli doesn't appear to support docker contexts currently, I flipped the order of options passed to the docker cli to ensure that any DOCKER_HOST environment variable will be used (if it is defined).

This change has been made in this pr and once merged will be pushed out in the next abctl release.

After this change has been released, you will be able to set the DOCKER_HOST environment variable to point to the correct docker location to use with the abctl tool.

[babaMar]

Thanks @colesnodgrass

[juliechoong]

Hi all. I'm running Airbyte OSS via docker compose in a Digital Ocean droplet. Recently I've tried migrating to abctl following the migration guide, but always end up with a 404 not found error by nginx, from abctl version 0.8.2 up until the recent version 0.11.0. I ran abctl local status after the abctl local install --migrate command and the output was fine. Not sure what went wrong that I keep getting the 404 error.

[jony1993]

@juliechoong is it now working for you?

[PieterBos94]

Since you are going to deprecate docker it seems better to directly move it to kubernetes instead of abctl right? I am currently running using docker on compute engine in GCP and running abctl on that doesn't seem like this best move. Are there any migration guides from docker to kubernetes? I am having trouble finding one.

[kev-datams]

Very interested for the same migration guide from GCP Compute Engine VM to Kubernetes 🙏
NB: we are already using an external DB on Cloud SQL

[abhisheksingh94]

Will the Docker Compose method before deprecation also be available for the v1 of the Airbyte platform update? I just wanted to ask because we are currently using Docker Compose and will continue to do so until our organization gets approval for Kubernetes deployment in the live environment. Therefore, we would like to have v1 of Airbyte platform update with the Docker Compose method available.

[briankariuki]

I think you should still keep docker compose. Not everyone is using kubernetes, also I couldn't get abctl to work on my local mac computer. Docker works fine

[kev-datams]

Still no way to make abctl work on a GCP VM with an external Cloud SQL database on our side
Probably a misconf with CloudSQLProxy on the K8s network 🤔
If you have more documentation, we would be happy to migrate

[marcosmarxm]

@airbytehq/platform-deployments fyi.

[mferreiracosta]

I'm trying to orchestrate airbyte with airflow. Has anyone managed to do it with local abctl and airflow with docker compose? And with both tools in Kubernetes?

[marcosmarxm]

I developer the Airflow Operator and tested using local abctl and Airbyte Cloud.