Almost nothing under "Deploy Airbyte" leads to a working installation using Helm on K8s.

[jesperbagge]

Page URL

No response

Description

Hello. I have for three days tried to deploy Airbyte 1.1.0 on K8s using the official documentation and have failed monumentally. Almost every aspect of the documentation to deploy Airbyte is outright wrong.

State and Logging Storage

For S3, the documentation details secret keys that should be named s3-access-key-id and s3-secret-access-key and that I should invoke their values using the key global.storage.storageSecretName. This breaks the Helm parser. According to an archived thread on Slack it should instead be global.storage.secretName. At least when using this key the parser doesn't break here. However the Helm deployment will instead fail due to that Secret.stringData.AWS_ACCESS_KEY_ID and Secret.stringData.AWS_SECRET_ACCESS_KEY has unknown objects of type "nil". It seems that the supplied secrets never gets picked up. Adding Secret.stringData.AWS_ACCESS_KEY_ID and Secret.stringData.AWS_SECRET_ACCESS_KEY to my K8s secrets does not help.

There is also a sidenote in the documentation that if I want to use another S3-compatible interface, then the endpoint-key should be supplied. The documentation does not mention where this key should be supplied. Perhaps it should be under global.storage.s3 but I can't test that since the deployment fails anyhow.

Secret Management

According to the documentation the four supported secret managers are AWS Secrets Manager, Google Secrets Manager, Azure Key Vault and Hashicorp Vault. However there are only configuration examples of the first three. I asked ChatGPT if it knew how to configure Hashicorp Vault as a secrets manager for Airbyte. It gave a suggestion that looked promising but the deployment failed due to Helm parsing errors.

External database

The documentation first outlines how to disable the internal Postgres db by setting postgresql.enabled: false. This will make the Helm parser fail, complaining about "nil" in Secret.stringData.DATABASE_PASSWORD and Secret.stringData.DATABASE_USER.

Skipping the setting postgresql.enabled: false will allow the deployment to continue but will make the airbyte-bootloader pod fail because it cannot connect to the external database. After a lot of trial and errors and printenv inside the failing pod I came to the conclusion that the only way to get at least something to work is to set the env DATABASE_URL to something like jdbc:postgresql://external-db.com:5432/airbyte. That was the only way to make the bootloader care about anything else than references to a K8s-internal service.

It still ultimately fails, though because the temporal pod fails to connect to a database. Probably another reference somewhere that doesn't get set properly.

All in all. The current state of the documentation and Helm chart is a mess.

[kagodarog]

@jesperbagge, you need to add these keys s3-secret-access-key and s3-access-key-id to the YOUR_NAMESPACE/airbyte-secrets secret if using S3 storage. This worked for me.

[jesperbagge]

@kagodarog i added those keys to the secret airbyte-config-secrets exactly like the documentation have in its example. Doesn’t do anything.

[kagodarog]

@jesperbagge, It's not that secret. At least with the latest helm version 1.1.0 there's a secret named airbyte-secrets that you need to update.

[remisalmon]

I am running into the same failures with the documentation of custom integrations (https://docs.airbyte.com/deploying-airbyte/#4-optional-for-customized-installations-only-customize-your-deployment) being wrong.

The secrets in airbyte-config-secrets do not get picked at all, either for external database or s3.

There is that new airbyte-secrets secret that is not documented, setting the global.database.user and global.database.password values directly in the helm chart seems to populate that secret instead, but the values are not propagated to DATABASE_USER and DATABASE_PASSWORDin the airbyte-bootloader pod which fails to boot.

[jesperbagge]

@jesperbagge, It's not that secret. At least with the latest helm version 1.1.0 there's a secret named airbyte-secrets that you need to update.

Thanks! Will try this at work tomorrow. This however further proves the mess that the docs / chart is in.

[jesperbagge]

@remisalmon setting DATABASE_USER, DATABASE_PASSWORD and DATABASE_URL is the only way I’ve managed to get the airbyte-bootloader not to fail. But Temporal isn’t picking up on those either so the deployment still fails. This is all very far from what’s in the documentation.

[sepsi77]

@jesperbagge I successfully deployed Airbyte to GKE and using Cloud SQL for database, cloud storage for logs and secret manager for secrets. Here are the relevant bits from my Terraform file. I'm not sure if it's significantly different for AWS. In addition to this, in the values.yaml I'm setting postgresql.enabled: false

Configure log and state storage

set {
name = "global.storage.type"
value = "GCS"
}

set {
name = "global.storage.storageSecretName"
value = kubernetes_secret.airbyte_config_secret.metadata[0].name
}

set {
name = "global.storage.bucket.log"
value = google_storage_bucket.airbyte_logs.name
}

set {
name = "global.storage.bucket.state"
value = google_storage_bucket.airbyte_logs.name
}

set {
name = "global.storage.bucket.workloadOutput"
value = google_storage_bucket.airbyte_logs.name
}
set {
name = "global.storage.gcs.projectId"
value = var.gcp_project_id
}
set {
name = "global.storage.gcs.credentialsPath"
value = "/secrets/gcs-log-creds/gcp.json"
}

Configure secret manager

set {
name = "global.secretsManager.type"
value = "googleSecretManager"
}

set {
name = "global.secretsManager.secretManagerSecretName"
value = kubernetes_secret.airbyte_config_secret.metadata[0].name
}

set {
name = "global.secretsManager.googleSecretManager.projectId"
value = var.gcp_project_id
}
set {
name = "global.secretsManager.googleSecretManager.credentialsSecretKey"
value = "gcp.json"
}

Configure external database

set {
name = "global.database.type"
value = "external"
}
set {
name = "global.database.host"
value = google_sql_database_instance.airbyte.private_ip_address
}
set {
name = "global.database.port"
value = "5432"
}
set {
name = "global.database.database"
value = google_sql_database.airbyte_db.name
}
set {
name = "global.database.secretName"
value = "airbyte-config-secrets"
}
set {
name = "global.database.userSecretKey"
value = "database-user"
}
set {
name = "global.database.passwordSecretKey"
value = "database-password"
}

[jesperbagge]

@sepsi77 Congratulations to a successful deployment. However, many of the yaml keys that you are setting in your Terraform file are exactly the same that is in the documentation and has also been proven not to work when deploying version 1.1.0 using Helm.
As an example, setting global.storage.storageSecretName will break the Helm parser and just specifying the external database by setting global.database.host will not get picked up by the airbyte-bootloader. But all these keys might have worked in the past.

[sepsi77]

@jesperbagge I've deployed the version 1.1.0.

[jesperbagge]

@sepsi77 well perhaps it is Terraform or GKE that does things differently than plain Helm and Kubernetes then, since the parser currently breaks with those config values.

[bgroff]

@jesperbagge can you post your values.yaml and secret.yaml file with the sensitive data stripped?

[bgroff]

Attached is a secrets.yaml file and values.yaml file that I have successfully used to deploy Airbyte 1.1.0 on AWS, using RDS, S3, and Secrets Manager:

Secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-password: "airbyte"

  # AWS S3 Secrets
  s3-access-key-id: <MY_ID>
  s3-secret-access-key: <MY_SECRET_KEY>

  aws-secret-manager-access-key-id: <MY_ID>
  aws-secret-manager-secret-access-key: <MY_SECRET_KEY>

values.yaml

postgresql:
  enabled: false

global:
  database:
    type: external

    # -- Secret name where database credentials are stored
    secretName: "airbyte-config-secrets"

    # -- The database host
    host: "airbyte.aws-id.us-west-2.rds.amazonaws.com"

    # -- The database port
    port: 5432

    # -- The database name
    database: "airbyte"

    # -- The database user
    user: "airbyte"

    # -- The key within `secretName` where password is stored
    passwordSecretKey: "database-password"

  storage:
    type: "S3"
    secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.
    bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: "us-west-2" ## e.g. us-east-1
      authenticationType: credentials ## Use "credentials" or "instanceProfile"

  secretsManager:
    type: awsSecretManager
    secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.
    awsSecretManager:
      region: us-east-2
      authenticationType: credentials ## Use "credentials" or "instanceProfile"

[jesperbagge]

Yes, I get the same error. I suspect that the error I pasted in my last message is due to the state and log store.

[bgroff]

and if you do a kubectl get secret airbyte-config-secrets -n airbyte you see the secret and the correct keys are set? It looks like the secret is found but the keys are not matching or are not set.

[jesperbagge]

This is what I get when running kubectl -n airbyte get secret airbyte-config-secrets -o yaml:

apiVersion: v1
data:
  database-password: <base64 snip>
  s3-access-key-id: <base64 snip>
  s3-secret-access-key: <base64 snip>
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"airbyte-config-secrets","namespace":"airbyte"},"stringData":{"database-password":"<snip>","s3-access-key-id":"<snip>","s3-secret-access-key":"<snip>"},"type":"Opaque"}
  creationTimestamp: "2024-11-13T07:26:39Z"
  name: airbyte-config-secrets
  namespace: airbyte
  resourceVersion: "391371065"
  uid: 525e48fc-056c-4b70-96d4-3a2c3cb1546d
type: Opaque

[bgroff]

And you are using the latest chart version? Can you use the secret and values file I posted above and see if those work for you (I have used them to deploy 1.1.0 so they should work). There is nothing about your values file that stands out as incorrect, but I cannot reproduce the issue you are having.

[jesperbagge]

The last try I did with a deployment was actually with a clean copy from your files, substituted with my values ofc. I even did a try with external storage as the only customization, as I noted a few posts above this one. Still breaks with error validating data: [unknown object type "nil" in Secret.stringData.AWS_ACCESS_KEY_ID.

[auguste-elax]

hey @jesperbagge, I ran into an issue similar to yours while upgrading airbyte to 1.2.0 from 0.67 with helm where the temporal pods would end in a crashloop state. If you're using postgres as an external database which enforces ssl, turns out you have to change the env variables names from SQL_XXX to POSTGRES_XXX.

Hope this helps!

    - name: POSTGRES_TLS
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_HOST_VERIFICATION
      value: "false"

[jesperbagge]

Hi @auguste-elax ! Thank you for the info. However, I’m not upgrading and there’s nothing in the documentation that supports this solution.

[bgroff]

The issue with the documentation has been addressed and I cannot reproduce this issue. There is a full working example that you can base your solution off of. At this time I am going to close this issue.

[jesperbagge]

Hi @bgroff . Well, this was a very deflating outcome. I was hoping that the open source community was beyond the "it works on my machine"-type of response.

I've had a couple of private DM:s on the Airbyte slack channel asking if I managed to resolve my issues. I've tried to deploy from a computer that didn't even have helm or kubectl installed, so I had to set those up first to really make sure that I didn't have anything weird on my own laptop that killed the deployment process.

May I ask what your thoughts on the error messages that I have been posting are? What do you make of it? Do you have any ideas on why the parser breaks on NULL-values on secrets that should be populated with the values from the secrets.yaml-file from your own example?

The user @remisalmon suggests in this reply that the only secret that gets picked up is another one called airbyte-secrets that is not mentioned in the documentation. When running a working vanilla deployment I can see that many values are installed in a secret named airbyte-airbyte-secrets. Can this be a lead into figuring out what is going on?

I just really want all this to work with an external database safely backed up.

[bgroff]

I provided a full working example. I used that example to deploy 1.2.0. Please rebase your deployment off of that values and secret file. It works. If you have issues with that file, then it is likely that your values have issues.

[joel-cant-wbt]

@jesperbagge did you ever get this to work, i am having an almost identical issue on AWS. An extra secret is being created and i am fairly sure that airbyte is ignoring the configuration / secrets

[jesperbagge]

@joel-cant-wbt unfortunately not. I’m not getting any help other than “it works on my machine” and none of my questions answered. I’m trying to deploy an exact copy of the supplied example and failing but it seems impossible to get anyone to accept that. And closing the discussion isn’t exactly helpful either. I’m close to giving up on this.