abctl migrate from docker compose

[tassilucas]

Hi,

We are currently running Airbyte v0.50.31 using the deprecated Docker Compose deployment. We are now trying to upgrade to a newer Airbyte version without recreating all connections, sources, and destinations.

According to the documentation, this should be possible using abctl with the --migrate flag:
https://docs.airbyte.com/platform/deploying-airbyte/migrating-from-docker-compose

However, it seems that the --migrate flag has since been deprecated:
#65071

Given this, what is the currently recommended migration path from Docker Compose to a newer Airbyte version?

We also tried migrating by performing database dumps between instances, but the schemas appear to differ significantly between versions, which makes this approach unreliable.

Any guidance or best practices would be greatly appreciated.

Thanks!

[ian-at-airbyte]

Hey @tassilucas. To get to the latest release (2.0) from where you are now, you're skipping ~100 releases, 10 of which were major post-1.0 releases. Airbyte's changed A LOT since 2023, and I don't think I recommend anyone attempt a straight upgrade over so many versions like this.

What's probably viable:

Option 1 (best): A fresh install. Use the API to export your configurations, then use the new API or Terraform to create the same configurations in the fresh install.

Option 2 (riskier): Incremental upgrades. What might work best is this (if memory serves correctly):

1. Get to the last version that supported Docker Compose. I think that's 0.64.7.

2. Install version 0.18 of abctl through the binary (this was the version available when 1.0 was released). https://github.com/airbytehq/abctl/releases/tag/v0.18.0

3. Use the migrate flag to migrate from Docker Compose to Airbyte 1.0.

4. Upgrade to version 0.30.2 of abctl

5. Use the upgraded abctl to migrate from Airbyte 1.0 to 2.0. Even this you might want to do in stages, as there were a few breaking changes between 1.0 and 2.0. Those are documented here: https://docs.airbyte.com/release_notes/breaking-changes

The main hump is getting from pre-1.0 to post-1.0. Once you hit 1.x, the underlying tech stabilized a lot more.

What's not viable:

• A database copy (for the reasons you suggest)

[tassilucas]

Hi @ian-at-airbyte, thanks for your response!

We needed to do some cleanup and apply fixes to our configuration, so we decided to create a new instance in a test environment and manually reconfigure everything. After validating that everything was working correctly, we created a production instance with the same version and loaded the database dump from the test environment into the production one.