Relax restriction of preferences to top-level packages

[staticfloat]

When preferences were first added, we originally did not have any preference merging across load path [0]. We later added that [1], but retained the requirement that for each individual element in the load path, preferences must have an entry in Project.toml listing the relevant package. This was partly on purpose (immediately binding the name to the UUID prevents confusion when a UUID<->name binding is not readily apparent) and partly just inheriting the way things worked back when preferences was written with just a single Project.toml in mind.

This PR breaks this assumption to specifically allow an entry in the Julia load path that contains only a single LocalPreferences.toml file that sets preferences for packages that may or may not be in the current environment stack. The usecase and desire for this is well-motivated in [2], but basically boils down to "system admin provided preferences that should be used globally". In fact, one such convenient method that now works is to drop a LocalPreferences.toml file in the stdlib directory of your Julia distribution, as that is almost always a part of a Julia process's load path.

[0] #37595
[1] #38044
[2] JuliaPackaging/Preferences.jl#33

[staticfloat]

There will doubtlessly be tests that must be adjusted as we're breaking some assertions that I had previously made on purpose, but I think it's clear that users want this feature, so we should just do it.

With this PR you should be able to have preferences that refer to packages that are not a top-level dependency in your current project (but that exist within a Manifest somewhere in your load path). This means that you can now have preferences applied to non-top-level deps, but also that you can have a LocalPreferences.toml with no accompanying Project.toml. You can even put a LocalPreferences.toml file in your stdlib/ directory for truly global preference-setting. Please feel free to leave comments on this PR about whether it works for you or not. There's likely many tests that need to be adjusted, but my Julia time is pretty limited these days, so if anyone else wants to help with fixing those tests, that'd be great!

[3f6a]

Could something break if I have two packages that have the same name but different UUIDs ?

[staticfloat]

Yes, that is what this check is intended to find.

[3f6a]

Couldn't LocalPreferences.toml specify UUID ? At least for the global preferences usage intended here

[staticfloat]

Yes, we could have made that decision back when we first introduced the format, but we decided against it because (1) it is intended to sit in the same place as a Project.toml and Manifest.toml, and (2) needing to enter UUIDs is not very user-friendly.

[3f6a]

we could have made that decision back when we first introduced the format

Can it be introduced now, without it being a breaking change?

[staticfloat]

Can it be introduced now, without it being a breaking change?

One could potentially do something like make a special _uuid top-level preference for a package, but I think that's a little ugly and the lookup should just be done in the corresponding Manifest, as for this preference to have an effect at all, the package must be in the manifest. We do not yet have widespread name pollution from competing registries, we mostly have one big registry (General) and smaller, private registries that typically don't shadow names from General. So in practice, the issues being guarded against here do not yet exist.

[3f6a]

like make a special _uuid top-level preference for a package

I do not understand why it can't just be called UUID? After all UUID is not used currently in LocalPreferences.toml, so it's available?

[staticfloat]

I do not understand why it can't just be called UUID?

Because every key under the package heading is assumed to be a preference, so this would conflict with a package that desires to set a key called UUID.