Integrate/expose go-libp2p resource manager

[BigLep]

go-libp2p v18 is shipping with a resource manager - yahoo! This issue encompasses the work to fully land this as a feature exposed to go-ipfs users. This includes:

• Integrating go-libp2p (i.e., doing the dependency update): feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680:
  • Estimate: 1 - needs review & cleanup – wip @lidel – see feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 (comment)
• Creating a command for resource manager stats: create a command for resource manager stats #8722
  • Covered in the PR above feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 – see feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 (comment)
• Expose stats to prometheus: feat: export rcmgr metrics to prometheus #8785 (will be easier once have done feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680):
  • Estimate: .5
• Make it self-service for go-ipfs users to configure the resource manager limits (Lotus example: improve resource manager integration filecoin-project/lotus#8318) . – see feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 (comment)
• For now use $IPFS_PATH/limit.json file + implicit defaults, not guaranteeing a contract. We want to learn about ergonomics from testing in production. In the end we may switch to Swarm.ResourceMgr.Limits config that works the same as the output of ipfs swarm limit commands.
  • see feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) #8680 and ResourceMgr UX improvements #8858
  • Estimate: 1
• Implement ipfs swarm limit [scope] --reset
  • tracked in Implement ipfs swarm limit [scope] --reset #8918 (not a blocker, could be in best-effort track for now)
• Expose Prometheus metrics from ResourceManager: feat: Expose Prometheus metrics from the libp2p ResourceManager #9385
• Testing in production (and fixing bugs):
  • Dashboard changes
  • Bootstrappers
  • Clusters
  • Gateway staging
  • Gateway banking https://github.com/protocol/bifrost-infra/issues/1815
    • https://github.com/protocol/bifrost-infra/pull/1832
    • dashboard
  • Estimate: 8 (high uncertainty - potential dragon 🐉 )
  • 0.17.0-rc1: ResourceMgr defaults sabotage DHT routing in Desktop/Brave nodes #9406
  • 0.17.0-rc1: ResourceMgr defaults clash with AcceleratedDHTClient #9405
  • Cleanup bifrost configuration: https://github.com/protocol/bifrost-infra/issues/1815#issuecomment-1316651157
• Cleanup
  • remove limits.json support? (see how big of a pain this is for libp2p-maintainers if we remove it)
  • process any remaining UX work from ResourceMgr UX improvements #8858
  • process any remaining UX work from Improve resource manager UX #9001
  • enable resource manager by default after selecting some default resource limits. We don't want to rely on implicit defaults from go-libp2p: Discussion: ResourceManager defaults when active by default #9322
  • Confirm that the system scope limits protect nodes (and not just the peer scope limits) per feat: Improve ResourceManager UX #9338 (comment)
• Config documentation and Release notes about this feature:
  • Estimate: 1
  • Pre on-by-default: Need to add a reminder: "if you are using internal config flags, these are the risks..."
  • On by default: This should show up in https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.17.md
    • Doc improvements and changelog for resource manager #9413

Total 2022-04-01 estimate: 12 (rounded up from 11.5)

Note: doing the configuration part will be easier once config is moved back into go-ipfs: ipfs/go-ipfs-config#151

[BigLep]

Assigned to @marten-seemann currently since he is doing some of the initial work. He'll ultimately need a go-ipfs partner for landing these changes and tieing up anything else.

[BigLep]

@guseggert : I put you as the owner now give you're doing the long-pole work of finding any landmines from production deployments. This should be prioritized first, and while collecting data we can work on #8858

[BigLep]

2022-05-12 conversation: we're looking good. Deployments to other banks is happening in https://github.com/protocol/bifrost-infra/issues/1815.

Resolving for now. We'll engage if Bifrost raises any issues as they continue to deploy.

[lidel]

@BigLep I assumed we are reopening this due to:

• bifrost issue: https://github.com/protocol/bifrost-infra/issues/1815
• UX gaps documented in Improve resource manager UX #9001
• a PR @guseggert will be working on to switch Swarm.ResourceMgr.Enabled flag to be disabled by default

lmk if I missed anything

[BigLep]

2022-09-27:

• Power user can configure anything
• By default, libp2p limits should scale based on system resources (similar to https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/README.md#usage )
• Need to decide what assertions make.

https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go#L21

[ajnavarro]

@BigLep may the notion link be wrong?

[BigLep]

@ajnavarro : doh - fixed - https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/README.md#usage

[BigLep]

2022-09-29:

• Adjust log levels so see whenever there is a resource checkout
• Expose grafana dashboards from go-libp2p resource manager: https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/obs/grafana-dashboards/README.md
• Write out what we're planning to do for when resource manager is enabled by default (Discussion: ResourceManager defaults when active by default #9322 )

[BigLep]

@ajnavarro : FYI that I've been updating the checklist and status based on what I know. I just added two items:

1. Confirm that the system scope limits protect nodes (and not just the peer scope limits) per feat: Improve ResourceManager UX #9338 (comment)
2. On by default: This should show up in https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.17.md

[ajnavarro]

@BigLep see my comment here related to point 1: #9338 (comment)

[BigLep]

Reported issues we need to address:

1. 0.17.0-rc1: ResourceMgr defaults sabotage DHT routing in Desktop/Brave nodes #9406
2. 0.17.0-rc1: ResourceMgr defaults clash with AcceleratedDHTClient #9405

[ajnavarro]

@BigLep see PR #9407

[BigLep]

@ajnavarro : for changelog and other doc updates, see #9413 . I know you were going to take the changelog, but I jumped in here while thinking about other doc improvements we should make. Feel free to make any changes and merge the PR so the RC can be cut.

Also, I think we Bifrost Gateway configuration should simplify to this: https://github.com/protocol/bifrost-infra/issues/1815#issuecomment-1316651157

[BigLep]

@ajnavarro : I assume this is possible but I also don't recall seeing it: how does someone learn what limits are actually passed to libp2p. I know we have ipfs swarm stats all to get current usage, but I want to be able to see what the limits are. I don't see any logging for it in https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr.go#L32

[ajnavarro]

@BigLep ipfs swarm limit all

[BigLep]

@ajnavarro : thanks. I made a docs update here: #9421

[BigLep]

@ajnavarro : I created a tracking issue for the critical followups we need to do: #9442

Please go ahead and edit/update with the workstreams.