how to deploy a custom connector to a different airbyte instance

[DaanDuppen]

Hi all,

I've been experimenting with OSS Airbyte and I'm loving it so far! I've set up a VM running airbyte and built a couple of custom connectors using the connector builder. This all works like a charm.

Now I'd like to run 2 VMs (dev and prod) so I can create/update my connectors in dev and deploy them to prod using a ci/cd pipeline. I'd expect this to be easy: using an airbyte API to get the custom connector yaml definition from my dev airbyte VM and use that to create/update a custom connector on my prod airbyte VM. Then copying the connection from my dev airbyte instance to my prod airbyte instance. The problem is that I can't find a way to do this.

I've tried a couple of methods:

• The airbyte api shows a way to create a source based on a custom connector, but I can't seem to find a way to get or create a custom connector using this api
• Airbyte terraform also seems to have the same problem
• The airbyte configuration api seems to have more endpoints, I haven't been succesful here either. Also it seems this api will be depricated

Am I missing something? Or is this simply not possible at the moment?
I'd love to hear what others are doing for their deployments of Airbyte OSS with custom connectors.

Thanks in advance!

[natikgadzhi]

You have a few ways to make your source available to another instance.

1. If your source is for a public API, and you made the source in Builder, you can contribute the source to Airbyte repo, and it will be built and made available in the catalog. In Airbyte 0.64+, there's Contribute option in Publish button. I'd recommend doing this, that's the easiest path forward.
2. You could build a custom connector image locally based on the exported manifest.yaml file from Builder, and then push it to Dockerhub and use it as a custom connector on the other instance.

To build a custom image, the easiest way is to build it in manifest-only format. It's new, and documentation is behind. Take a look at source-calendly as an example in our repo.

Basically:

1. Export the manifest, rename it to manifest.yaml.
2. Clone airbyte repo, go to its root.
3. Copy an existing manifest-only connector to a new connector directory, for example cp -rf airbyte-integrations/connectors/source-calendly airbyte-integrations/connectors/source-your-thing.
4. Move your manifest to that directory, replace the original connector manifest.
5. Edit metadata.yaml file in that directory, make sure the docked image name maps to your connector. Also change the version and such. And allowedHosts.
6. Cleanup README in that dir.
7. You're ready to build! Build it as you would build any custom connector in airbyte repo, I.e. airbyte-ci connectors --name source-your-thing build.

Good luck!