GitSync invalid key format in /etc/git-secret/ssh

[laserpedro]

Official Helm Chart version

1.10.0

Apache Airflow version

2.6.2

Kubernetes Version

1.30

Helm Chart configuration

    persistence:
      subPath: dags/platform
      size: 10Gi
      storageClassName: default-efs
    gitSync:
      enabled: true
      repo: [email protected]:my-org/dag-repo.git
      branch: main
      rev: HEAD

      sshKeySecret: airflow-git-sync-secret

Docker Image customizations

No response

What happened

I have created a deploy ssh key in the repo to enable gitsync to only access this repo in particular with read only access. I generate a ssh key locally and updated the public key on the deploy key. The private key is stored in AWS secrets manager as a plain text file and I let external secret operator create a secret that by construction is base64 encoded and matches the format indicated by the documentation:

    # If you are using an ssh clone url, you can load
    # the ssh private key to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: airflow-ssh-secret
    #   data:
    #     # key needs to be gitSshKey
    #     gitSshKey: <base64_encoded_data>

However I receive the error:

"msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b main --depth 1 [email protected]:my-org/dag-repository.git /git): exit status 128: { stdout: "", stderr: "Cloning into '/git'...\nWarning: Permanently added 'github.com,20.X.X.X.X' (ECDSA) to the list of known hosts.\r\nLoad key "/etc/git-secret/ssh": invalid format\r\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." }" "failCount"=1

I have create d a debug pod with the secret mounted as it is on the airflow-worker pod and the pem file looks fine for me:
cat /etc/git-secret/ssh

-----BEGIN OPENSSH PRIVATE KEY-----

-----END OPENSSH PRIVATE KEY-----

What you think should happen instead

No response

How to reproduce

Deploy Airflow on kubernetes 1.30 and deploy the 1.10 airflow helm chart trying to connect using a deploy ssh key to the dag folder.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This is not an airflow issue - you have some problem with format, but you should debug it. I suggest you login to the container and try regular git command in it and try to fix the problem (that's what I'd do at least). Also there are more issues in the log (permission denied) which is likey the root cause of the problem. It needs a bit of debugging that I think only you can do - so I convert it to discussion here - maybe someone else who made similar configuration problem will be able to help you in the discussion.

[laserpedro]

@potiuk thank you for your answer and indeed it is more of a discussion. Trust me I debugged it a lot before opening this, not in my attitude to throw to maintainers whatever issues we can have by using open source projects ;)

Actually the thing is that I added to the secret the key known_hosts and now it is working, wanted to know if someone noticed the same. thanks

[sspaeth-r7]

Hi, I wanted to chime in and say I'm experiencing the exact same problem. I verified my key format was not messed up by diffing the key that works on on my laptop locally with the one that the Helm chart mounted to /etc/git-secret/ssh in the container in my cluster. Both were identical.

I'm using the dags.gitSync.sshKey property in my values.yaml.

Actually the thing is that I added to the secret the key known_hosts and now it is working, wanted to know if someone noticed the same. thanks
By this, do you mean you added the host entry to your known_hosts? If so, I added the same entry that works locally on my machine to the dags.gitSync.knownHosts value and that unfortunately didn't fix it for me.

Does anyone have any more ideas as to what could be causing this?

Thanks in advance!

[sspaeth-r7]

Ok, I figured out what the issue was for me and posting in case it's the cause of anyone else's pain and suffering.

TL;DR my secret which gets passed as the dags.gitSync.sshKey Helm value is just the key itself, without a newline character at the end. For some reason SSH breaks down and gives a completely unhelpful error just because the file doesn't include an invisible EoF \n character.

I'm using Terraform to pass values, and so I ended up doing the following which fixed it:

sshKey = sensitive("${var.ssh_key}\n")

This formatting issue didn't come up in testing because when I copied the text out of the K8s volume and pasted it into my local machine's Vim, it automatically added the newline at EoF, because Vim. That created the illusion that there were no issues with the key being mounted in the container.

[potiuk]

😱 🙀

[potiuk]

Thanks for sharing - that is one of those "tales" to share - how I spent two days chasing missing EOL issue that vim usage made incredibly difficult to find.