hostRules for private Docker Hub stopped working

[calvis]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

I had the following config which successfully bumped private images hosted in Docker Hub:

  "hostRules": [
    {
      "hostType": "docker",
      "matchHost": "https://index.docker.io/v2/theorgwiththeimages",
      "username": "thebotuser",
      "password": "{{ secrets.DOCKERHUB_PASSWORD }}"
    }
  ],

This was using the actual user password. My intent for the matchHost was to use the bot user exclusively for the images in theorgwiththeimages, and to use the default Renovate credentials for all other Docker Hub images.

Recently, this stopped working, despite not having made any changes to the config:

Please correct - or verify that you can safely ignore - these dependency lookup failures before you merge this PR.

Failed to look up docker package theorgwiththeimages/theprivateimage

From what I can tell from the Dependency Dashboard issue revision history, this failure message appeared on Sept 16th. Based on the timing, it could be related to this announcement: https://www.docker.com/blog/deprecation-of-password-logins-on-cli-with-docker-sso-enforcement/

Most of the time, there's no helpful information in the logs, it just has a DEBUG entry that it failed, with no other surrounding information:

DEBUG: Failed to look up docker package theorgwiththeimages/theprivateimage
{
  "dependency": "theorgwiththeimages/theprivateimage"
  "packageFile": ".circleci/config.yml"
}

I saw a failed HTTP request one time in the logs, but I cannot reproduce it, even by rerunning with exactly the same config that was present for the job in question:

DEBUG: Response code 401 (Unauthorized)
...
"body": {
        "details": "your account must log in with a Personal Access Token (PAT) - learn more at docs.docker.com/go/access-tokens"
      },

I'm worried that there is caching involved that is preventing me from seeing why authentication is failing except for once in a blue moon. Is it possible the failure itself has been cached?

I would rather be using a token, and I've tried to get a token to work many times (both when we originally set up the hostRules and recently, using the new "Host rules" in the organization settings). I've tried both sending it as a token and sending username/password with the token as the password. I can successfully docker login using the token as the password and pull the image in question so I know it is valid. I've tried just about every value for matchHost I can think of, including docker.io. I've tried to use exactly the config from this discussion answer, but it fails without detailed debug information "Failed to look up docker package" as above.

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks