Loading of a git source is slow due to manifest parsing

[epage]

When you have some git dependencies or patches, loading them takes a long time

For the traces from #14238, notice how slow the first resolve is compared to the second. Almost all of that is taken up in Manifest parsing

Note: this slow down is most relevant for tooling like rust-analyzer or rust-defined completions when it may be called in interactive contexts without any compilation happening.

Related: #13724, #14603

[epage]

Previously discussed at https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Redundant.20code.20in.20.60GitSouce.60.3F

[epage]

The core problem is that git sources are indexed by name. To find a name, we walk the source to find all Cargo.toml files except if they are in a submodule or a hidden directory except if a Cargo.toml has a path dependency in it in an ignored location.

Without that second exception, we could parse Cargo.toml to toml::Table, look up the name, and be done. That would cut out probably 60% of the manifest parsing performance hit.

With the second exception, we have to do a full parse of the Cargo.toml file.

Ideas

• Allow dependencies to specify the path within the repo (Git dependency with a path into a sub-folder location #11858)
  • Needs design work
  • Requires people to opt-in to get performance gain
• Store the path in Cargo.lock
  • Also helps with ensuring consistent path to ambiguous name
  • Potential overlapping design work with Git dependency with a path into a sub-folder location #11858 (source id, package id, package id spec)
  • Investigation is needed whether we can get the information from Cargo.lock to the source to be use
• Store a cache in .cargo-ok
  • Content in the file shouldn't cause checkout thrashing
  • Being in that file keeps the checkout self-contained (delete it and everything is gone) and avoids fingerprinting issues with old cargo (they ignore this file)
  • Likely should only have one entry (thrashing on version changes) as a simple reclamation policy
  • Either fingerprint cargo like we do rustc or use the git commit sha if its not dirty, not caching when it is dirty
• Tell people to avoid ambiguous names, or test files (only reduces part of the problem)
• A file in the repo that indexes it (require opt-in by the dependency)
• A .cargoignmore file in the repo for what not to index (requires opt-in by the dependency)
• Speed up parsing by caching immutable Cargo.tomls in a faster format (likely cut 1/6 of parse time out)

[epage]

Main risk: there aren't enough manifests for us to skip loading