Standardising on pnpm

[sndrs]

Proposal

pnpm should be our standard package manager.

Why do we need a standard?

Different package manager have different behaviours in terms of how they manage range resolution, store deps and report issues to consumers.

They also have different levels of support from third parties. Standardising on one tool minimises the amount of hoop-jumping and investigation of problematic idiosyncrasies.

Why pnpm?

• it's hoisting strategy (i.e. do not hoist) means we know we're consuming/bundling the dependencies we think we are
• it's extremely fast
• it has great error messaging
• it's under active development
• it's robust and has an existing non-trivial user base (this proposal was prompted by @arelra highlighting vercel as the latest high-profile adopters)
• it's working very well in https://github.com/guardian/csnx

Why not?

• dependabot does not support pnpm, and [does not plan to as of this quarter], but...(Feature request: support the pnpm package manager dependabot/dependabot-core#1736 (comment))
  • renovate does support pnpm and in many other ways is superior, but has only tentative support of the tech council
  • it's already used in https://github.com/guardian/csnx
• snyk does not support it out of the box, but...
  • there's https://github.com/snyk-tech-services/snyk-pnpm-github-action – @kenoir is this a viable option if we otherwise wanted to standardise on pnpm? no

[kenoir]

https://github.com/snyk-tech-services/snyk-pnpm-github-action has been archived and Snyk are not committing to support pnpm so I don't think it's an option for us.

[kenoir]

Have we thought about the long term viability of pnpm? Will it still be around (or the best thing) in a year or 2? Compared to the long term viability of plain old npm?

[sndrs]

pnpm is on version 7 and predates yarn, and merged 373 PRs in the last year, so I would say it's under active development.

aside from general performance issues, the problem is that npm still hoists packages, which is the behaviour we are trying to move away from.

they do tend to adopt features from 3rd party installers (eventually), so they may drop hoisting at some point. but for now npm is slower and doesn't solve the hoisting issue.

(npm has always felt to me like a best-efforts package manager – node needs something to get you going with package registries but it's never been as good as 3rd party options (once they appeared)).

[kenoir]

pnpm is on version 7 and predates yarn

Sounds alright to me then.

[mxdvl]

PNPM is also supported by Node’s Corepack:

Corepack is an experimental tool to help with managing versions of your package managers. It exposes binary proxies for each supported package manager that, when called, will identify whatever package manager is configured for the current project, transparently install it if needed, and finally run it without requiring explicit user interactions.

From Snyk, the package health scores are also quite telling:

• npm 100/100
• pnpm 98/100
• yarn 85/100

[mxdvl]

PNPM just outpaced yarn in number of downloads this week. https://npm-stat.com/charts.html?package=pnpm&package=yarn&from=2021-01-01&to=2023-06-01

[kenoir]

Vulnerability monitoring for pnpm is supported by https://github.com/google/osv-scanner#input-a-lockfile.

[mxdvl]

Dependabot are about to look at it: dependabot/dependabot-core#1736 (comment)

[sndrs]

just fyi, dependabot beta support for pnpm is enabled on csnx, and we've switched over from renovate in #543.

all looking good so far (minus some nice functionality in renovate, like batching PRs)

[sndrs]

pnpm is now supported by dependabot! dependabot/dependabot-core#1736 (comment)

[sndrs]

hold those horses – except for security updates (c/o @joecowton1) dependabot/dependabot-core#7434

[jorgeazevedo]

Reading on the documentation about pnpm's hoisting strategy, it says

When installing dependencies with npm or Yarn Classic, all packages are hoisted to the root of the modules directory. As a result, source code has access to dependencies that are not added as dependencies to the project

So if I understand this correctly, pnpm prevents situations where a project can run on my machine because all its dependencies are installed in node_modules, but fails elsewhere because a dependency was not declared in package.json?

[sndrs]

@jorgeazevedo exactly - node module resolution algorithm doesn't check versions of packages or origins, just whether it can find something in a node_modules directory in the current dir tree with the correct name.

you can make your own node_modules folder and import from it without using a package manager - package manager's just populate a folder that follows a naming convention (node_modules).

not hoisting prevents inadvertent use of transitive dependencies and ensures each file imports the version of a dep specified in the package it belongs to (rather than whatever wins in the hoisting negotiation.

when bundling for browsers this can mean shipping multiple copies of the same lib, but the alternative is shipping a single copy that is then incompatible with the code that imports it (project still need to manage this situation)

[sndrs]

here's a good, very in-depth analysis of the problem https://github.com/npm/rfcs/blob/main/accepted/0042-isolated-mode.md

[sndrs]

just a bit more context - used in azure devops and across MS dependabot/dependabot-core#1736 (comment)

[joecowton1]

Dependabot now supports vulnerability scanning with pnpm: https://github.blog/changelog/2023-08-02-pnpm-support-for-dependency-graph-dependabot-alerts-and-dependabot-security-updates/

[sndrs]

is this the last thing stopping us making it a formal recommendation?

[joecowton1]

We need clarification from Security that they're happy with Dependabot coverage as a Snyk alternative and that we won't loose it if we end up not paying for GHE. I've asked @akash1810 for clarification.

[akash1810]

Keen to avoid myself being a bottleneck, so cc @guardian/devx-security.

[sndrs]

DCR is now on pnpm (as of guardian/dotcom-rendering#9476), with vulns handled by dependabot.

@guardian/devx-security are aware and on board.

Let's maybe let the dust settle on it, and if all looks good in a couple of weeks, I think we can formalise a recommendation.

[joecowton1]

Thanks Alex, this is real progress!

[mxdvl]

Worth noting that pnpm’s handling of transitive dependencies makes is very difficult and unreliable to update them:

• pnpm update -r --latest <package-name> Updates all dependencies instead of only updating the one I selected, (MONOREPO) pnpm/pnpm#8154
• pnpm up -i --latest Updates unspecified packages pnpm/pnpm#8031
• pnpm dedupe is behaving nondeterministically pnpm/pnpm#8155
• pnpm up doesn't update peerDependencies pnpm/pnpm#8081

Furthermore, identifying dependencies with pnpm why is difficult, as the results are silently truncated.