Gitlab v4 api 404 errors on commit status update - Wrong URLs vs. API spec?

[kyrian666]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

I'm not absolutely sure if this is an issue with my config, with atlantis, or the underlying xanzy/go-gitlab module. I could be mistaken on any aspect of this as it's my first foray into Atlantis, and I have had a hard time making it work with the age of the infrastructure and code I am trying to make it work with.

Pointing Atlantis at a modern gitlab installation using API v4 fails consistently on the commit status updates.

I believe it is because the URL form used in the API calls that Atlantis makes against gitlab to update commit/MR statuses is incorrectly formed.

I've ruled out authorisation, project name escaping, and other possible trouble spots, I've even asked gitlab support and they concur with my assessment that atlantis is hitting the wrong URLs.

My client is extremely security sensitive so I have had to redact all this heavily, but I think you get the gist from the URL structure in the logs.

Compare with the gitlab v4 API specification: https://docs.gitlab.com/ee/api/commits.html#commit-status

Because I can't rule out that the failure to update the git status makes atlantis grind to a halt I can't be sure that my pre workflow hooks are at fault either. And there is no debugging output of the pre-workflow hooks that I could find which makes debugging much harder than it might otherwise be.

Also the debug output of Atlantis does not necessarily seem to reflect exactly what the underlying xanzy/go-gitlab module is actually hitting, down to the HTTP request method level, which made things harder still.

Reproduction Steps

Push any old gitlab MR at a gitlab v4 API instance connected to atlantis. Doesn't seem to matter the content.

Logs

Mar 05 12:48:02 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:02.726Z","caller":"events/working_dir.go:330","msg":"ran: git clone --depth=1 --branch BAU-test-commit-hook --single-branch https://MYATLANTISUSER:@MYGITLABSERVER.MYDOMAIN.COM/MYORG/MYPROJECT.git /home/atlantis/.atlantis/repos/MYORG/MYPROJECT/270/default. Output: Cloning into '/home/atlantis/.atlantis/repos/MYORG/MYPROJECT/270/default'...","json":{}}
Mar 05 12:48:02 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:02.727Z","caller":"events/pre_workflow_hooks_command_runner.go:121","msg":"Processing pre workflow hook 'Initialise git submodules before atlantis starts the terraform init otherwise it breaks.', Command 'plan', Target commands [plan]","json":{"repo":"MYORG/MYPROJECT","pull":"270"}}
Mar 05 12:48:02 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:02.727Z","caller":"events/pre_workflow_hooks_command_runner.go:129","msg":"Running pre workflow hook: 'Initialise git submodules before atlantis starts the terraform init otherwise it breaks.'","json":{"repo":"MYORG/MYPROJECT","pull":"270"}}
Mar 05 12:48:02 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:02.974Z","caller":"vcs/gitlab_client.go:442","msg":"GET /projects/MYORG/MYPROJECT/merge_requests/270 returned: 200","json":{}}
Mar 05 12:48:02 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:02.974Z","caller":"vcs/gitlab_client.go:411","msg":"Head pipeline found for merge request 270, source 'external'. refTarget 'BAU-test-commit-hook'","json":{}}
Mar 05 12:48:03 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:03.064Z","caller":"vcs/gitlab_client.go:434","msg":"POST /projects/MYORG/MYPROJECT/statuses/e18a4852a5fa74d2bae3cf8afa6d6e19REDACTED returned: 404","json":{}}
Mar 05 12:48:03 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"warn","ts":"2024-03-05T12:48:03.064Z","caller":"events/pre_workflow_hooks_command_runner.go:147","msg":"unable to update pre workflow hook status: GET https://MYGITLABSERVER.MYDOMAIN.COM/api/v4/projects/MYORG/MYPROJECT/statuses/e18a4852a5fa74d2bae3cf8afa6d6e19REDACTED: 404 {error: 404 Not Found}","json":{"repo":"MYORG/MYPROJECT","pull":"270"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).runHooks\n\t/home/runner/work/atlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:147\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).RunPreHooks\n\t/home/runner/work/atlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:90\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:193"}
Mar 05 12:48:03 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"error","ts":"2024-03-05T12:48:03.064Z","caller":"events/command_runner.go:196","msg":"Error running pre-workflow hooks GET https://MYGITLABSERVER.MYDOMAIN.COM/api/v4/projects/MYORG/MYPROJECT/statuses/e18a4852a5fa74d2bae3cf8afa6d6e19REDACTED: 404 {error: 404 Not Found}.","json":{"repo":"MYORG/MYPROJECT","pull":"270"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:196"}
Mar 05 12:48:03 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"error","ts":"2024-03-05T12:48:03.064Z","caller":"events/command_runner.go:199","msg":"'fail-on-pre-workflow-hook-error' set, so not running plan command.","json":{"repo":"MYORG/MYPROJECT","pull":"270"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:199"}
Mar 05 12:48:03 MYCICDSERVER.MYDOMAIN.COM atlantis[2538301]: {"level":"debug","ts":"2024-03-05T12:48:03.065Z","caller":"metrics/debug.go:52","msg":"timer","json":{"name":"atlantis.cmd.autoplan.execution_time","value":1.003306532,"tags":{},"type":"timer"}}

Environment details

Standard latest atlantis downloaded from github.

Run with: /usr/local/bin/atlantis --config /etc/atlantis-server.yaml --repo-config /etc/atlantis-repo.yaml server

Atlantis server-side config file:

url: http://MYCICDSERVER.MYDOMAIN.COM/
gitlab-hostname: "MYGITLABSERVER.MYDOMAIN.COM"
gitlab-user: "MYATLANTISUSER"
gitlab-token: "glpat-REDACTED"
gitlab-webhook-secret: "REDACTED"
repo-allowlist: "MYGITLABSERVER.MYDOMAIN.COM/MYORG/MYPROJECT"
log-level: "debug"
fail-on-pre-workflow-hook-error: true
web-basic-auth: true
web-username: MYBASICAUTHUSER
web-password: MYBASICAUTHPASS
automerge: false
version: 3
projects:
  dir: .
#  workspace: default
  terraform_version: "0.11.15-oci"
  autoplan:
    enabled: false
  autoplan_modules: false

Repo atlantis.yaml file:

repos:
- id: MYGITLABSERVER.MYDOMAIN.COM/MYORG/MYPROJECT
  allowed_overrides: [workflow]
  allow_custom_workflows: true
  pre_workflow_hooks:
    - run: git submodule init
      description: Initialise git submodules before atlantis starts the terraform init otherwise it breaks.
      shell: bash
      shellArgs: -x
      commands: plan
    - run: git submodule update --recursive --remote
      description: Update git submodules before atlantis starts the terraform init otherwise it breaks.
      shell: bash
      shellArgs: -x
      commands: plan
    - run: REDACTED STEP
      description: REDACTED
      shell: bash
      shellArgs: -x
      commands: plan
  branch: /.*/
  plan_requirements: [mergeable]
  apply_requirements: [mergeable, approved, undiverged]
  import_requirements: [mergeable, approved, undiverged]
  workflow: perftest-legacy
workflows:
  perftest-legacy:
    plan:
      steps:
      - env:
          name: https_proxy
          value: 'REDACTED'
      - env:
          name: no_proxy
          value: 'REDACTED'
      - env:
          name: PLAN
          value: 'true'
      - run:
          command: A.SHELL.SCRIPT.PATH
    apply:
      steps:
      - env:
          name: https_proxy
          value: 'REDACTED'
      - env:
          name: no_proxy
          value: 'REDACTED'
      - run: APPLY=true A.SHELL.SCRIPT.PATH

It's a straight up ubuntu 20.04 instance with nothing of note in the sphere of atlantis.

This current strange setup with pre-workflow hooks is necessary to cope with git submodules because Atlantis doesn't seem to do so natively. There are other horrors in there but I don't think they ever even come into play because the pre-workflow hooks fail due to the API URLs being wrong.

Additional Context

[kyrian666]

Issue #2685 also makes suggestions of a change to the gitlab API's around statuses so this might be a duplicate or related problem?

[Jonomir]

We have a similar issue. Started happening when upgrading from v0.28.5 to v0.29.0

Here are our logs

atlantis-0 atlantis {"level":"info","ts":"2024-09-02T14:45:03.155Z","caller":"events/events_controller.go:656","msg":"parsed comment as command=\"plan\" verbose=false dir=\"\" workspace=\"\" project=\"\" policyset=\"\", clear-policy-approval=false, flags=\"\"","json":{"repo":"company/group/repo","pull":123}}
atlantis-0 atlantis {"level":"info","ts":"2024-09-02T14:45:03.472Z","caller":"events/events_controller.go:699","msg":"Running comment command 'plan' on repo 'company/group/repo', pull request: 123 for user 'user.name'.","json":{"repo":"company/group/repo","pull":123}}
atlantis-0 atlantis {"level":"warn","ts":"2024-09-02T14:45:04.713Z","caller":"events/pre_workflow_hooks_command_runner.go:76","msg":"unable to update plan commit status: 404 Not Found","json":{"repo":"company/group/repo","pull":"123"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).RunPreHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:76\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:339"}
atlantis-0 atlantis {"level":"warn","ts":"2024-09-02T14:45:05.350Z","caller":"events/pre_workflow_hooks_command_runner.go:144","msg":"unable to update pre workflow hook status: 404 Not Found","json":{"repo":"company/group/repo","pull":"123"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).runHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:144\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).RunPreHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:339"}
atlantis-0 atlantis {"level":"info","ts":"2024-09-02T14:45:05.350Z","caller":"events/pre_workflow_hooks_command_runner.go:145","msg":"is api? false","json":{"repo":"company/group/repo","pull":"123"}}
atlantis-0 atlantis {"level":"info","ts":"2024-09-02T14:45:05.350Z","caller":"events/pre_workflow_hooks_command_runner.go:147","msg":"is api? false","json":{"repo":"company/group/repo","pull":"123"}}
atlantis-0 atlantis {"level":"error","ts":"2024-09-02T14:45:05.350Z","caller":"events/command_runner.go:342","msg":"Error running pre-workflow hooks 404 Not Found.","json":{"repo":"company/group/repo","pull":"123"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:342"}
atlantis-0 atlantis {"level":"error","ts":"2024-09-02T14:45:05.350Z","caller":"events/command_runner.go:345","msg":"'fail-on-pre-workflow-hook-error' set, so not running plan command.","json":{"repo":"company/group/repo","pull":"123"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:345"}
atlantis-0 atlantis {"level":"warn","ts":"2024-09-02T14:45:06.104Z","caller":"events/command_runner.go:351","msg":"unable to update plan commit status: 404 Not Found","json":{"repo":"company/group/repo","pull":"123"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:351"}

[matzegebbe]

We are experiencing the same problem with GitLab and version 0.29.0. Strangely, everything works if we open a new merge request (and close the old one beforehand). Is this also the case for you @Jonomir

In our case, pre_workflow_hook_runner.go only triggers local commands why atlantis checks the gitlab api in that state? I found #2685 (comment) what could explain the error

with 'fail-on-pre-workflow-hook-error' set, there is also no feedback to the gitlab comments

logs

{
  "level": "warn",
  "ts": "2024-09-27T11:20:54.828Z",
  "caller": "events/pre_workflow_hooks_command_runner.go:76",
  "msg": "unable to update plan commit status: 404 Not Found",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).RunPreHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:76\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:339"
}
{
  "level": "warn",
  "ts": "2024-09-27T11:20:55.513Z",
  "caller": "events/pre_workflow_hooks_command_runner.go:144",
  "msg": "unable to update pre workflow hook status: 404 Not Found",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).runHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:144\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPreWorkflowHooksCommandRunner).RunPreHooks\n\tgithub.com/runatlantis/atlantis/server/events/pre_workflow_hooks_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:339"
}
{
  "level": "info",
  "ts": "2024-09-27T11:20:55.513Z",
  "caller": "events/pre_workflow_hooks_command_runner.go:145",
  "msg": "is api? false",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  }
}
{
  "level": "info",
  "ts": "2024-09-27T11:20:55.513Z",
  "caller": "events/pre_workflow_hooks_command_runner.go:147",
  "msg": "is api? false",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  }
}
{
  "level": "error",
  "ts": "2024-09-27T11:20:55.513Z",
  "caller": "events/command_runner.go:342",
  "msg": "Error running pre-workflow hooks 404 Not Found.",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:342"
}
{
  "level": "error",
  "ts": "2024-09-27T11:20:55.513Z",
  "caller": "events/command_runner.go:345",
  "msg": "'fail-on-pre-workflow-hook-error' set, so not running plan command.",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:345"
}
{
  "level": "warn",
  "ts": "2024-09-27T11:20:56.432Z",
  "caller": "events/command_runner.go:351",
  "msg": "unable to update plan commit status: 404 Not Found",
  "json": {
    "repo": "/my/repo-obfuscate",
    "pull": "62"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:351"
}

[fede-r1c0]

I had this problem, I couldn't find a solution. I rolled back to v0.28.5 of atlantis for the moment. I would appreciate if there is any update on the bug. GitLab Premium 17.7

[m0ps]

Got the same issue as @fede-r1c0 has. I spent a day trying to understand what was wrong until I came to this thread. And after downgrading to v0.28.5 - all ok.

[matzegebbe]

I may have found a workaround: We added a noop job in GitLab. Since we have more than just the external pipelines from Atlantis, it looks like this resolved the issue with the spinning pipeline status. We were wondering why another Atlantis repository never failed, and it turned out it had local linters configured, etc.

gitlab-ci.yml:

stages:
  - noop

empty_job:
  stage: noop
  script:
    - echo "This job does nothing but atlantis runs into 404 https://github.com/runatlantis/atlantis/issues/4313"

[m0ps]

Hah... it helped, @matzegebbe... Much appreciate. So, there should be either a real pipeline or a placeholder (like you suggested).

[m0ps]

It seems this (#5033) PR is supposed to fix this issue.