Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flatten/de-dupe npm lockfile #2079

Closed
XhmikosR opened this issue Nov 25, 2019 · 10 comments
Closed

Flatten/de-dupe npm lockfile #2079

XhmikosR opened this issue Nov 25, 2019 · 10 comments
Labels
F: language-support Issues specific to a particular language or ecosystem; may be paired with an L: label. L: javascript:npm npm packages via npm T: feature-request Requests for new features

Comments

@XhmikosR
Copy link

So, this situation has bit me numerous times :)

Basically, after a while using dependabot, one ends up with many duplicate packages. A recent example is twbs/bootstrap#29730

I'm not sure how dependabot could mitigate this issue. npm dedupe does not seem to solve the issue. On the other hand, regenerating the lock file shouldn't be done by default since it can break many things.

I know there is dependabot/feedback#313, but it seems that was focused mostly on yarn.
@feelepxyz feelepxyz changed the title flatten lockfile Flatten/de-dupe npm lockfile Nov 27, 2019
@feelepxyz
Copy link
Contributor

@XhmikosR Thanks for submitting! I've attempted to fix this in the past but didn't even get the npm dedupe command working properly as it sometimes ended up breaking the lockfile. I think this should be solved in npm itself because it probably requires some internal changes.

We're super stretched at the moment so don't have much spare capacity to work on wider improvements for the next few months.

@XhmikosR
Copy link
Author

@feelepxyz NP, I understand. :)

Like I said, I don't even know how you could fix it on your side either. I will try to raise an issue with npm

@XhmikosR
Copy link
Author

XhmikosR commented Dec 7, 2019

@feelepxyz which command does dependabot run? npm i package@version?

@feelepxyz
Copy link
Contributor

@XhmikosR yes pretty much but Dependabot doesn't call the CLI directly, instead it uses the library functions to avoid downloading node_modules and calling life-cycle hooks: https://github.com/dependabot/dependabot-core/blob/master/npm_and_yarn/helpers/lib/npm/updater.js#L20

@XhmikosR
Copy link
Author

XhmikosR commented Dec 9, 2019

I wonder if the same issue happens when running npm i package@version...

@XhmikosR
Copy link
Author

XhmikosR commented Jun 2, 2020

This is starting to be very annoying :/

https://github.com/twbs/bootstrap/pull/30951/files

I have to redo the lock file every couple of weeks.

@infin8x infin8x transferred this issue from dependabot/feedback Jun 29, 2020
@infin8x infin8x added F: language-support Issues specific to a particular language or ecosystem; may be paired with an L: label. L: javascript:npm npm packages via npm T: feature-request Requests for new features labels Jul 20, 2020
@yudai-nkt
Copy link

Any updates on this?

I received a dependabot PR with a patch release update breaking the CI (yudai-nkt/jupyterlab_city-lights-theme#103), and it turned out to be caused by duplicated packages thanks to jupyterlab/jupyterlab#9699. npm dedupe didn't fix my problem neither, so I thought it's related to this issue rather than a new issue.

@yudai-nkt yudai-nkt mentioned this issue Jan 29, 2021
Closed
jcscottiii added a commit to web-platform-tests/wpt.fyi that referenced this issue Aug 26, 2022
@jcscottiii
Move all dependency updates to dependabot
0bbdc06 
Remove renovate and use dependabot for go, python and node
This is to serve as research

Proposal:

There are some notable differences between renovate and dependabot. If
we want to continue with using only dependabot, this is what I would
suggest.

1. Setup dependabot for go, python, node and docker. Make sure it does not auto merge.
2. Make the deployment step require an approval for dependabot https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
  - Need to verify that it can be limited at the per bot level

More notes down below
---

Warnings

There are some differences between dependabot and renovate:
There are things renovate can do that dependabot can't do:
1. Run npm dedupe to keep the lock files pretty clean.
  [Notes](dependabot/dependabot-core#2079)
2. Group updates together github/roadmap#148

Extra work needed specifically for dependabot

1. Auto merging is possible but extra work needs to be done.
   [Notes](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request)
  - I don't think auto merging is a good idea.
2. Need to create a GCLOUD_KEY_FILE_JSON secret specifically for dependabot to use. Make sure it has the necessary scopes and no more.

Misc notes

- For go dependencies - It does indded run go mody tidy. (which is absolutely necessary) [Notes](https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/)
- Dependabot has a docker updater too. We should use that.
jcscottiii added a commit to web-platform-tests/wpt.fyi that referenced this issue Aug 26, 2022
@jcscottiii
Move all dependency updates to dependabot
797a936 
Remove renovate and use dependabot for go, python and node
This is to serve as research

Proposal:

There are some notable differences between renovate and dependabot. If
we want to continue with using only dependabot, this is what I would
suggest.

1. Setup dependabot for go, python, node and docker. Make sure it does not auto merge.
2. Make the deployment step require an approval for dependabot https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
  - Need to verify that it can be limited at the per bot level

More notes down below
---

Warnings

There are some differences between dependabot and renovate:
There are things renovate can do that dependabot can't do:
1. Run npm dedupe to keep the lock files pretty clean.
  [Notes](dependabot/dependabot-core#2079)
2. Group updates together github/roadmap#148

Extra work needed specifically for dependabot

1. Auto merging is possible but extra work needs to be done.
   [Notes](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request)
  - I don't think auto merging is a good idea.
2. Need to create a GCLOUD_KEY_FILE_JSON secret specifically for dependabot to use. Make sure it has the necessary scopes and no more.

Misc notes

- For go dependencies - It does indded run go mody tidy. (which is absolutely necessary) [Notes](https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/)
- Dependabot has a docker updater too. We should use that.
@jcscottiii jcscottiii mentioned this issue Aug 26, 2022
Merged
jcscottiii added a commit to web-platform-tests/wpt.fyi that referenced this issue Aug 26, 2022
@jcscottiii
Move all dependency updates to dependabot
c539d5f 
Remove renovate and use dependabot for go, python and node
This is to serve as research

Proposal:

There are some notable differences between renovate and dependabot. If
we want to continue with using only dependabot, this is what I would
suggest.

1. Setup dependabot for go, python, node and docker. Make sure it does not auto merge.
2. Make the deployment step require an approval for dependabot https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
  - Need to verify that it can be limited at the per bot level

More notes down below
---

Warnings

There are some differences between dependabot and renovate:
There are things renovate can do that dependabot can't do:
1. Run npm dedupe to keep the lock files pretty clean.
  [Notes](dependabot/dependabot-core#2079)
2. Group updates together github/roadmap#148

Extra work needed specifically for dependabot

1. Auto merging is possible but extra work needs to be done.
   [Notes](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request)
  - I don't think auto merging is a good idea.
2. Need to create a GCLOUD_KEY_FILE_JSON secret specifically for dependabot to use. Make sure it has the necessary scopes and no more.

Misc notes

- For go dependencies - It does indded run go mody tidy. (which is absolutely necessary) [Notes](https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/)
- Dependabot has a docker updater too. We should use that.
jcscottiii added a commit to web-platform-tests/wpt.fyi that referenced this issue Aug 26, 2022
@jcscottiii
Move all dependency updates to dependabot
3156fab 
Remove renovate and use dependabot for go, python and node
This is to serve as research

Proposal:

There are some notable differences between renovate and dependabot. If
we want to continue with using only dependabot, this is what I would
suggest.

1. Setup dependabot for go, python, node and docker. Make sure it does not auto merge.
2. Make the deployment step require an approval for dependabot https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
  - Need to verify that it can be limited at the per bot level

More notes down below
---

Warnings

There are some differences between dependabot and renovate:
There are things renovate can do that dependabot can't do:
1. Run npm dedupe to keep the lock files pretty clean.
  [Notes](dependabot/dependabot-core#2079)
2. Group updates together github/roadmap#148

Extra work needed specifically for dependabot

1. Auto merging is possible but extra work needs to be done.
   [Notes](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request)
  - I don't think auto merging is a good idea.
2. Need to create a GCLOUD_KEY_FILE_JSON secret specifically for dependabot to use. Make sure it has the necessary scopes and no more.

Misc notes

- For go dependencies - It does indded run go mody tidy. (which is absolutely necessary) [Notes](https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/)
- Dependabot has a docker updater too. We should use that.
@jcscottiii jcscottiii mentioned this issue Aug 29, 2022
Closed
jcscottiii added a commit to web-platform-tests/wpt.fyi that referenced this issue Nov 23, 2022
@jcscottiii
Move all dependency updates to dependabot (#2945)
81b9aa1 
* Move all dependency updates to dependabot

Remove renovate and use dependabot for go, python and node
This is to serve as research

Proposal:

There are some notable differences between renovate and dependabot. If
we want to continue with using only dependabot, this is what I would
suggest.

1. Setup dependabot for go, python, node and docker. Make sure it does not auto merge.
2. Make the deployment step require an approval for dependabot https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
  - Need to verify that it can be limited at the per bot level

More notes down below
---

Warnings

There are some differences between dependabot and renovate:
There are things renovate can do that dependabot can't do:
1. Run npm dedupe to keep the lock files pretty clean.
  [Notes](dependabot/dependabot-core#2079)
2. Group updates together github/roadmap#148

Extra work needed specifically for dependabot

1. Auto merging is possible but extra work needs to be done.
   [Notes](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request)
  - I don't think auto merging is a good idea.
2. Need to create a GCLOUD_KEY_FILE_JSON secret specifically for dependabot to use. Make sure it has the necessary scopes and no more.

Misc notes

- For go dependencies - It does indded run go mody tidy. (which is absolutely necessary) [Notes](https://github.blog/changelog/2020-10-19-dependabot-go-mod-tidy-and-vendor-support/)
- Dependabot has a docker updater too. We should use that.

* add commented docker section

* disable deployment for dependabot PRs

* add quotes
@jeffwidman
Copy link
Member

👋 Sorry for the slow response, we're finally getting a chance to look at some of our backlog.

This is a really old issue... is it still happening or can this be closed?

Dependabot generally tries to shell out to npm / yarn native package managers, so it's unclear if the root problem here is in Dependabot or upstream in the package managers themselves... especially if npm dedupe isn't resolving it.

@XhmikosR
Copy link
Author

XhmikosR commented Feb 9, 2023

Maybe this is now fixed assuming you are using newer npm, it's been some time since I last noticed it.

@jeffwidman
Copy link
Member

Sounds good, thanks for letting us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
F: language-support Issues specific to a particular language or ecosystem; may be paired with an L: label. L: javascript:npm npm packages via npm T: feature-request Requests for new features
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@feelepxyz @XhmikosR @jeffwidman @infin8x @yudai-nkt