[BUG] npm hangs on gitlab repo install

[nVitius]

Current Behavior:

I am installing a dependency from a private gitlab repo with git+ssh. Every time I install it, npm takes a really long time to complete the installation. I ran it with -ddd and noticed this:

npm http fetch GET 503 https://gitlab.com/users/sign_in 70361ms attempt #3
npm timing reifyNode:node_modules/<gitlab-package-name> Completed in 72435ms

Expected Behavior:

npm doesn't try to sign in to gitlab when installing an ssh package

Steps To Reproduce:

npm i --save git+ssh://gitlab.com/some_user/private_repo#master

Environment:

macOS 11.2.1
node 12.20.1
npm 7.5.4

[nVitius]

@ljharb @darcyclarke @isaacs
Just pinging you guys as I've seen you on some recent issues.

I would love to help debug this, but I don't know where to start. I looked through the codebase a bit, but couldn't find where this endpoint would be called. If you can point me in the right direction at least, I can try to work on this.

[nVitius]

@ljharb @nlf
Pinging you two as I've seen you comment on recent issues.

Please let me know if there's someone else that should be looking at this instead, or if more info is needed here. Thanks!

By the way, I tried this again with npm 7.6.0 and the issue is still occurring.

[dillonbartkus]

I'm having the same problem as well, trying to npm install an internal package.

[dillonbartkus]

@nVitius I don't know if this helps you, but what worked for me was to completely uninstall node / npm (I used this as a guide : https://stackoverflow.com/questions/11177954/how-do-i-completely-uninstall-node-js-and-reinstall-from-beginning-mac-os-x), then re-install node using the downloadable package. Hope this helps.

[nVitius]

@dillonbartkus I'm using node/npm with nvm. After you reinstalled, are you still using npm version 7.6.0?

[dillonbartkus]

@nVitius I picked LTS version, which uses NPM version 6.14.11

[nlf]

using 7.6.0, can you try the same command but with --loglevel=silly at the end and share the log?

[nVitius]

@nlf that's what -ddd does. I can post the whole log, but it's not very informative around that section in particular:
https://pastebin.com/rB3teabE

The relevant lines are 127-129. Here's a screencap:

It looks like this is only happening with Gitlab repos. Installing a Github repo with git+ssh does not cause this issue.

[nlf]

what do you get if you run:

git ls-remote ssh://[email protected]/wake_up_warror/warrior-vue-shared.git

in a shell? does it prompt you for anything?

[nlf]

if it does, and you accept the prompt, then a follow up npm install command should work as expected

additionally, if when you run the npm install command you see something like

The authenticity of host 'gitlab.com (2606:4700:90:0:f22e:fbec:5bed:a9b9)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.

even though we're hiding the line that tells you to press enter, if you press enter the installation will continue.

a way to prevent this from happening in the first place would be to

export GIT_SSH_COMMAND="ssh -oStrictHostKeyChecking=accept-new"

somewhere in your shell profile, which will automatically accept keys that are not in your ~/.ssh/known_hosts but will verify keys that are there have not changed.

[nVitius]

The ls-remote command runs successfully. It does not prompt for anything. My ssh-agent is set up correctly, so I don't expect any issues with the git commands themselves.

The npm module is actually installed correctly. The problem is that npm hangs while trying to connect to https://gitlab.com/users/sign_in. This adds over a minute to the install.

I'm fairly certain that this is an issue with npm.

[nlf]

I was able to reproduce it, but the above made the reproduction stop working. I’ll make a repo and test further

[nVitius]

Here is the package.json that I have in that repo:

{
  "name": "warrior-vue-shared",
  "version": "1.2.2",
  "description": "Shared Assets and Vue Components",
  "scripts": {
    "build": "babel src -d dist",
    "prepare": "npm run build"
  },
  "main": "dist/index.js",
  "files": [
    "assets",
    "dist"
  ],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@babel/cli": "^7.12.16",
    "@babel/core": "^7.12.16",
    "@babel/preset-env": "^7.12.16"
  },
  "peerDependencies": {
    "vue": "^3.0.5"
  }

I don't know if it's relevant, but the repo is under a "Group" in gitlab.

[nlf]

The group could be very relevant! I’ll do some more testing tomorrow and follow up here

[nlf]

good news: i can reproduce this issue now!
bad news: i'm still not entirely sure what's going on, but now that i can reliably reproduce the problem it should be fixable

[nlf]

ok, i believe i've determined a root cause.

what happens is when installing from a known git provider, even when you specify the url via ssh, we attempt to fetch a tarball from that provider first. in the case of gitlab, this redirects to a sign in page with a 503 status code.

we see that 503 and interpret it as a failure we can retry, so we apply an exponential backoff and retry again. and then again. this ultimately takes just over a minute.

a reasonable workaround would be to add --fetch-retries=0 to your command line, or running npm config set fetch-retries=0 to save it to your ~/.npmrc for all future commands. this will still fail once, but that happens relatively quickly and we would not retry or have the exponential backoffs in delays.

we're discussing whether there's a more permanent fix for this in the cli team

[nVitius]

@nlf thanks for looking into this.

I feel that Gitlab should be returning a 401 for these requests instead. A quick search on their issue tracker didn't bring up any existing threads about this; I'll create an issue with them to see if they would be open to that change.

Either way, I'll use the fetch-retries config option for now. Thanks for the suggestion.

Also, could you link to the code that handles this? I wasn't able to find it myself when I was originally looking into the issue.

[nlf]

thanks for following up with them about that, i tend to agree with you that the 503 they currently respond with isn't the right thing to do, but browser checks are weird critters. i'm hopeful we'll find that they agree and can change it to simplify this for everyone.

fetch-retries are handled in make-fetch-happen, we determine if a request can be retried here: https://github.com/npm/make-fetch-happen/blob/latest/index.js#L336-L345

the option is assigned to npm.flatOptions here: https://github.com/npm/cli/blob/latest/lib/utils/flat-options.js#L118

@npmcli/arborist initiates the request via pacote which uses npm-registry-fetch to apply some common conventions and passes through to make-fetch-happen where the request is made and retrying is actually handled. it's pretty confusing, we're working hard to consolidate and simplify a lot of areas of our code base and this is a chain of dependencies that's been on my mind for a while now.

i'm going to go ahead and close this issue since we have a reasonable workaround for you. thanks again for reaching out to gitlab about this!

[isaacs]

The root cause of this should be fully addressed (and the workaround thus unnecessary) when npm/pacote#65 is landed.

[JC3]

I feel that Gitlab should be returning a 401 for these requests instead. A quick search on their issue tracker didn't bring up any existing threads about this; I'll create an issue with them to see if they would be open to that change.

@nVitius Do you have a link to the issue you made?

[JC3]

@nlf Wouldn't another fix be to not try and fetch the tarball and instead try the URL that the user actually specified as the first attempt? If I want the tarball I can specify its URL...

[luanmuniz]

Are there any updates on a possible solution for this issue? I've been facing it for the last few days, using the fetch-retries flag, but it's still a big issue in my company. cc @nlf

[JC3]

@luanmuniz

Are there any updates on a possible solution for this issue? I've been facing it for the last few days, using the fetch-retries flag, but it's still a big issue in my company. cc @nlf

I don't know if this is helpful, but at my company we ended up avoiding this issue by using GitLab's built-in package registries, then instead of specifying the repo URL, we do proper package deployments to the registry and specify namespace + package name. So the issue no longer occurs since we're using packages now instead of git URLs. Perhaps that's a solution you could consider. It's very easy to set up and a lot easier to manage versioning, since you can use actual semver with the packages and all of npm's versioning features.

See https://docs.gitlab.com/ee/user/packages/package_registry/ for an overview. We went with the "use a project as a package registry" technique (as opposed to a monorepo) and just have a single project dedicated to being a package registry for everything else.

[arnabmaji]

ok, i believe i've determined a root cause.

what happens is when installing from a known git provider, even when you specify the url via ssh, we attempt to fetch a tarball from that provider first. in the case of gitlab, this redirects to a sign in page with a 503 status code.

we see that 503 and interpret it as a failure we can retry, so we apply an exponential backoff and retry again. and then again. this ultimately takes just over a minute.

a reasonable workaround would be to add --fetch-retries=0 to your command line, or running npm config set fetch-retries=0 to save it to your ~/.npmrc for all future commands. this will still fail once, but that happens relatively quickly and we would not retry or have the exponential backoffs in delays.

we're discussing whether there's a more permanent fix for this in the cli team

@nlf
Can we disable this behaviour so that npm does not fetch tarballs from known git providers like Gitlab even when the specified protocol is git+shh? We are actually running into rate-limiting issues due to this.