Cannot push (login) to Harbor

[greyarch]

Actual behavior
Kaniko cannot login to Harbor the same way the old kaniko could - with credentials in /kaniko/.docker/config.json.

Instead I get an error:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<my-harbor-url>/<my-project>/<my-image>:<image-tag>": POST https://<my-harbor-url>/<my-project>/<my-image>/blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: <my-project>/<my-image>, action: push: unauthorized to access repository: <my-project>/<my-image>, action: push

This happens at the beginning of the build process.

Expected behavior
I expect it to work as it used to. This is a regression.

To Reproduce
I am running this in a gitlab pipeline, using the martizih/kaniko:v1.25.5-debug image as a job image. This is because I need a shell for the gitlab job.

[mzihlmann]

Hi @greyarch, thank you for opening this issue.
Hmmm... could you help me triage the issue? we recently introduced env-credential helper. Maybe you could try to deactivate it with credential helpers option:

--credential-helpers=""

and then explicitly activate it with:

--credential-helpers="env"

By default all credential-helpers are activated.

[greyarch]

Hey @mzihlmann, thanks for the swift response.

Using the env credentials helper works, although I had to look in the code to see how to deal with dashes in the fqdn :)

It looks however, that the docker (/kaniko/.docker/config.json ) credentials provisioning does not, yet the documentation suggests that it should?

[mzihlmann]

yes config.json must work. my idea was to disable env credential helper to see whether it causes the regression.

Docs is a good point, will add them.

Apart from that it would be super helpful to see in which version the regression started. If it started in 1.25.5 that would put more weight on env credential helper as the suspect.

[greyarch]

Ah, I see. No, disabling the credential helpers did not help. Had the same issue.

[mzihlmann]

can you provide more details? How do you configure the config.json, maybe paste the script snippet you're using. Is it gitlab.com or a private instance? docker or kubernetes runners? Anything related in the environment, ie. KANIKO_...? what cli args are you passing?

I'm wondering whether it is a harbor specific issue as I can login to gitlab registry with config.json no problem. Let me set up a harbor instance to verify.

[greyarch]

This may well be Harbor specific.
As for Gitlab, I am only executing in a Gitlab job. From there I am trying to push to Harbor.
I am constructing the config.json by base64'ing the username:password and adding in to the auths attribute. I am pretty sure I am doing it right, because this used to work with the old Kaniko, before the fork.

[0hlov3]

Hi there,

i tested pushing to harbor with the latest debug and v1.25.5-debug.

1. Create the namespace

kubectl create ns development

2. Create the secret

kubectl create secret docker-registry -n development harbor-kaniko --from-file=auth.json

3, Start a debugging pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kaniko-dev
  namespace: development
  labels:
    app: kaniko-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kaniko-dev
  template:
    metadata:
      labels:
        app: kaniko-dev
    spec:
      containers:
        - name: kaniko
          image: martizih/kaniko:debug
          command: ["/busybox/sh", "-c", "tail -f /dev/null"]
          tty: true
          stdin: true
          volumeMounts:
            - name: docker-config
              mountPath: /kaniko/.docker/config.json
              subPath: config.json
              readOnly: true
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
      volumes:
        - name: docker-config
          secret:
            secretName: harbor-kaniko
            items:
              - key: .dockerconfigjson
                path: config.json
      restartPolicy: Always

4. Exec into the debugging Container and create a Dockerfile:

k exec -ti kaniko-dev-6554747f97-ckkwm -- /bin/sh
vi Dockerfile

5. Start the Build:

/kaniko/executor --dockerfile Dockerfile --destination redact.ed/kaniko/kaniko:testv1

The output:

INFO[0000] Retrieving image manifest alpine:3.22.2
INFO[0000] Retrieving image alpine:3.22.2 from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest alpine:3.22.2
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine:3.22.2' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN apk --no-cache add --update $APK_PACKAGES &&     rm -rf /var/cache/apk/* requires it.
[... a lot of other output ...]
INFO[0004] Pushing image to redact.ed/kaniko/kaniko:latest
INFO[0004] Pushed redact.ed/kaniko/kaniko@sha256:61137b79efe880170fbaf713ddc5e30a6d74cef20379c784da4027d3fc1b9d7f

EDIT:
With the martizih/kaniko:v1.25.5-debug:

/workspace # /kaniko/executor version
Kaniko version :  v1.25.5
workspace # /kaniko/executor --dockerfile Dockerfile --destination redact.ed/kaniko/kaniko:testv2
[...]
INFO[0003] Taking snapshot of full filesystem...
INFO[0003] CMD ["/bin/bash"]
INFO[0003] Pushing image to redact.ed/kaniko/kaniko:testv2
INFO[0004] Pushed redact.ed/kaniko/kaniko@sha256:281b1f802921b2b4b2d6363a8e713adfb0ccb987c41566f8da409728fbbaa732

EDIT2:
I was not able reproduce the error locally:

❯ podman run -ti --entrypoint='' -v $(pwd):/test martizih/kaniko:debug /bin/sh

/workspace # cp /test/auth.json /kaniko/.docker/config.json
/workspace # vi Dockerfile
/workspace # /kaniko/executor --dockerfile Dockerfile --destination redact.ed/kaniko/kaniko:testv1

I think we need more information to reproduce the error.

[mzihlmann]

I now also created a harbor installation, I cannot reproduce this issue. My current hypothesis is that the $ is interpreted by your shell? But that doesn't explain why it works in google's kaniko.

$ echo robot$library+blubb:password | base64 | base64 -d
robot+blubb:password

Instead you should escape the sign

$ echo robot\$library+blubb:password | base64 | base64 -d
robot$library+blubb:password

You can do so with single quotes too:

$ export HARBOR_USERNAME='robot$library+blubb'
$ export HARBOR_PASSWORD='password'
$ echo $HARBOR_USERNAME:$HARBOR_PASSWORD | base64 | base64 -d
robot$library+blubb:password

also make sure to use echo -n otherwise a newline gets added to your password

echo -n robot\$library+blubb:password | base64

[mzihlmann]

Hey @greyarch,
I would really like to debug this issue but so far I was not able to reproduce it in my setup. I have a few ideas around possible misconfigurations above but nothing that would explain a regression. Would you mind to share more information on your setup? Or even help with bisecting the issue?

[xadhoom]

Hello, sorry for writing on a closed issue, tell me if is better to open a new one.

I've exactly the same issue, and in my harbor username/password I don't have any shell char that can be misinterpreted.
My harbor usersname is something like robot-my.privateregistry.com and the pass has only standard chars in the range a-zA-Z0-9.

To build the docker config json the following is done in the CI:

echo "{\"auths\":{\"${REGISTRY_URL_PROD}\":{\"auth\":\"$(printf "%s:%s" "${REGISTRY_AUTH_USERNAME}" "${REGISTRY_AUTH_SECRET}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json

Then I tested using env vars, same result.

I suspect it's something with harbor setup, like others with google kaniko works as expected.

If you need to try some image, I'll happy to do it.

[mzihlmann]

Hi @xadhoom
Yes it would be awesome if you could help bisecting this issue. It's likely due to some library we bumped as chainguard is affected too and they literally didn't do anything but dependency bumps. I think once we triage which dependency, we can open an issue in their repo and it will be resolved quickly. Unfortunately we can't reproduce the issue ourselves so we must rely on users like you in gathering further information.

You could start by testing releases and then using git bisect to find the exact commit. We publish images for each commit so you don't need to build from source. Please reach out if you need help in running git bisect, I'm happy to do a pair-programming session to finally get to the bottom of this issue.

Thank you 🙏