Run abctl with 2 CPU & 4 GB of memory

[nataliekwong]

Today, abctl works on a machine with 4 or more CPU's and at least 8 GB of memory with the --low-resource-mode flag. You can pass the low resource mode flag when installing Airbyte with abctl:

abctl local install --low-resource-mode

You can see our suggested resources documentation here: https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#suggested-resources

This equivalents to a t2.large (Amazon EC2 T2 Instances) using --low-resource-mode. It does not yet work on t2.medium.

This discussion aims to collect interest and feedback on enabling Airbyte to sufficiently run on 2 or more CPu's and 4 GB of memory, or a t2.medium instance.

[techieshark]

Very interested. Thanks for suggesting.

I'm evaluating Airbyte by running locally on an M1 with 16GB ram and actually struggling to get it up and running with the following settings (though at this point I'm not sure if I'm still running into resource issues or just some other local config issues).

[caileyfitzgerald]

Yes, this would be a great help! Since switching to abctl from docker compose in our staging environment, we are seeing significantly longer load times (up to hours) for even small amounts of data (20 records), whereas docker compose is able to load magnitudes higher in a fraction of the time.

[ngohuuduc]

We face the same thing.

[jediwarpraptor]

Is it the # of CPUs or the Memory that's the bottleneck. I'm contemplating standing this up on a 2-processor 16GB VM.

[et3rn4ls]

Is it possible to change to low resource mode after installation? I deployed it on a m6a.xlarge machine (4 vCPUs / 16 GB) but had problems with lack of resources (FailedScheduling). Now it is running on a m6a.2xlarge (8 vCPUs / 32GB) but it is not viable due to costs.

[colesnodgrass]

You could trying running the install command again passing the --low-resource-mode flag. An install over an existing install will attempt to upgrade the cluster in place.

abctl local install --low-resource-mode [any other flag originally provided]

If that fails, you can always uninstall and and install again. Your data should be persisted during the uninstall process.

abctl local uninstall && abctl local install --low-resource-mode [any other flags required]

[brothatru]

I'm running on AWS EC2 r7g.xlarge (4 cpus x 32 gb ram), but got hit with the error:

Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again.

Setting the --low-resource-mode works in my case.

What exactly does the --low-resource-mode flag do?

Are there any other ways to "fine tune" resources using environment variables? In my case to maximize my 4 cpus and 32 gb ram?

• ie. JOB_MAIN_CONTAINER_CPU_LIMIT or JOB_MAIN_CONTAINER_MEMORY_LIMIT?

Asking because I don't want "low resource mode" to limit to, for example 2 cpus x 8gb of ram, when my machine has much more.

Thanks in advance!