Adopt XDG Base Directory Specification

[athas]

See design discussion in #680

Cabal/cabal-install now uses the XDG Base Directory Specification to store configuration, caches, and the store. Specifically,
$XDG_CONFIG_HOME/cabal stores the configuration file,$XDG_CACHE_HOME/cabal stores downloaded packages and similar, and $XDG_STATE_HOME/cabal mainly contains the store of compiled
packages.

The dist/dist-newstyle directories are not affected.

On Windows, these XDG paths are mapped to other (hopefully) appropriate locations. See the updated Cabal User Guide for details.

If the CABAL_DIR environment variable is set, the indicated directory will be used to store all Cabal-related files, as in
previous versions.

Backwards compatibility: If ~/.cabal already exists, this will be interpreted as CABAL_DIR=~/.cabal. This means that upgrading on an existing system already using cabal-install should not cause any change in behaviour. An existing system can be migrated by deleting ~/.cabal (possibly copying ~/.cabal/config to ~/.config/cabal/config first).

---

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.

Please also shortly describe how you tested your change. Bonus points for added tests!

[athas]

I have also found a few places where certain path computations seem to be duplicated instead of importing functions from Distribution.Simple.InstallDirs. Should I make small preliminary PRs to fix these in advance?

An example (where I have fixed it, I think): https://github.com/haskell/cabal/pull/7386/files#diff-ae0158c0ab0fa116a8e5deeea1f7a9809c9c061a1646444580df9b4c131263b0L66-L82

[phadej]

The tricky part is where to put /caba/store. It's kind of a cache, so XDG_CACHE_HOME makes sense, OTOH installed executables (and libraries for environment files) are there as well, so if cache is purged things are broken (the cache is not autopopulated).

I guess XDG_CACHE_HOME is not bad, but this limitation is good to mention.

---

I think that CABAL_DIR should go away with this change. making its default equal to XDG_CACHE_HOME isn't future proof. What if something belongs to XDG_DATA_HOME, or XDG_STATE_HOME (There is a single base directory relative to which user-specific state data should be written. - is this a place to put nix-style store? see also this SO q/a)

---

I stress, if this change goes in, there should be a very clear "migration" documentation. (Which at simplest is: continue using explicit config, set by CABAL_CONFIG pointing to old directories you use, or remove everything and start from scratch).

[phadej]

Please not comment here. This PR is draft, and discussion what it should do is in #680 (comment)

[Mikolaj]

I took the liberty of merging this branch with master, to kick off the CI.

[athas]

CI wrinkle: the Debian cabal is used to generate the config file (with v2-update), which is then put in $HOME/.cabal/config, but the tests will expect it in $HOME/.config/caba/config.

[gbaz]

Along with where cabal stores its own files, we should start using xdg for writing env files and other ghc interaction, cf for example: https://gitlab.haskell.org/ghc/ghc/-/issues/20684

[Mikolaj]

Also, see the cabal issue about that: #7819

[Mikolaj]

Hello! Can we help to push this forward?

[Mikolaj]

I see a decision is needed. Could somebody remind what the decision is about? We could discuss that on this Thursday's meeting (all devs are invited).

[athas]

I think the only real decision to be made is a set of instructions about where to put what, and whether CABAL_DIR should be used for anything except backwards compatibility. I didn't stop working on it because it's particularly difficult; only because I got distracted by other things. I recall only two meaningful technical challenges:

1. Backwards compatibility, which I think should be triggered if CABAL_DIR is set or ~/.cabal exists.
2. Some parts of the test suite use the CABAL_DIR environment variable and will need to use the XDG config stuff instead, but they can't depend directly on the cabal-install package for reasons I don't remember clearly anymore. This can probably be solved in ad-hoc ways, since it's just a test suite.

[gbaz]

It sounds like you can just continue to push forward, including keeping back-compat for CABAL_DIR for now. If there's any specific question about what goes where, please highlight it -- otherwise, it looks like everything is on track, and we're excited to see this proceed.

(in the documentation, we can note that we're discouraging future use of CABAL-DIR)

[Mikolaj]

(in the documentation, we can note that we're discouraging future use of CABAL-DIR)

which implies that the answer to your question "whether CABAL_DIR should be used for anything except backwards compatibility" is that it should not. We plan to gradually deprecate CABAL_DIR and remove it, helping its users to find good alternate workflows along the way.

[Mikolaj]

@athas: BTW, I've just had read you email about the XDG overhaul to haskell-cafe. Great job! I can't find it mentioned on https://discourse.haskell.org/ (and I haven't looked r/haskell nor on IRC/matrix). I think posting a link to that email and perhaps also pasting the contents would be a service to the community. What do you think?

[ulysses4ever]

it's also easy to submit a link to the announcement to HaskellWeekly: just PR https://github.com/haskellweekly/haskellweekly. @athas let us know if you need help with posting any of it. We really want to have people try it as much as possible before the release.

[athas]

I have posted to the Discourse. I'll see about submitting a link to Haskell Weekly News.

[ulysses4ever]

Good, thanks. I took liberty to post the Discourse link on Reddit: https://www.reddit.com/r/haskell/comments/y946bj/request_for_testing_of_the_xdgenabled_cabalinstall/

[mouse07410]

Here's my current ~/.cabal directory contents:

$ ll ~/.cabal
total 40
drwxr-xr-x   11 ur20980  staff    352 Sep  9 09:51 ./
drwxr-xr-x+ 385 ur20980  staff  12320 Oct 20 13:24 ../
drwxr-xr-x   51 ur20980  staff   1632 Oct 12 20:52 bin/
-rw-r--r--    1 ur20980  staff   5523 Sep  9 09:51 config
-rw-r--r--    1 ur20980  staff   5533 Mar 20  2022 config.backup
drwxr-xr-x    4 ur20980  staff    128 Jul 17 13:22 lib/
drwxr-xr-x   13 ur20980  staff    416 Sep  5 17:10 logs/
drwxr-xr-x    3 ur20980  staff     96 Sep 28  2021 packages/
drwxr-xr-x    4 ur20980  staff    128 Jul 17 14:39 share/
drwxr-xr-x    8 ur20980  staff    256 Sep  5 17:10 store/
-rw-r--r--    1 ur20980  staff     15 Sep 29  2021 world

After migration - where exactly would each of those entries go?
Note: special attention to ~/.cabal/lib/ and ~/.cabal/store/ directories, because that's what dynamically built libraries and executables use.

[ulysses4ever]

@mouse07410 thanks for trying it! I think, to get the new structure, you have to nuke ~/.cabal first.

[athas]

After migration - where exactly would each of those entries go?

Does this section in the documentation answer the question? If not, we should improve the documentation.

I think the documentation does not explain where the lib directory ends up (it's ~/.local/lib), but that's mostly because I'm a bit shaky on when exactly things are put in that directory, so I don't know how to describe it.

[mouse07410]

After migration - where exactly would each of those entries go?

Does this section in the documentation answer the question? If not, we should improve the documentation.

No, it doesn't, IMHO.

What I'd find satisfactory, would be an exhaustive list of the possible legal entries in the current ~/.cabal directory with the mapping of each to its new location.

I think the documentation does not explain where the lib directory ends up (it's ~/.local/lib), but that's mostly because I'm a bit shaky on when exactly things are put in that directory, so I don't know how to describe it.

And IMHO that's bad: not only will this collide with the Stack products (currently going to ~/.local/), but I'm aware of other apps utilizing that directory too. Are we heading towards "Cabal hell" all over again, but on a larger scale? Because cabal-v1, at the very least, did not impact other ecosystems, while this change will.

[athas]

What I'd find satisfactory, would be an exhaustive list of the possible legal entries in the current ~/.cabal directory with the mapping of each to its new location.

I don't know what "legal" means in this context, and I'm not sure cabal ever had an exhaustive list of such directories. While working on this PR, it seemed various parts of the code just asked for "the cabal dir" and then created whatever they needed as subdirectories. It would be possible to describe movement of the most common directories in the changelog entry, but I'm not sure it's useful in the user's guide itself.

And IMHO that's bad: not only will this collide with the Stack products (currently going to ~/.local/), but I'm aware of other apps utilizing that directory too. Are we heading towards "Cabal hell" all over again, but on a larger scale?

I don't think so, precisely because many other ecosystems have put stuff in that directory for years, and it doesn't seem to a widely reported problem. You only have conflicts when you have identically named programs installed by different package managers, which seems to be a rare case. And cabal at least makes it easy to change the installation directory in the configuration file. This concern was also brought up in the original issue: #680

[mouse07410]

I don't know what "legal" means in this context, and I'm not sure cabal ever had an exhaustive list of such directories

To me "legal" in this context means "what cabal itself is supposed to put there", as opposed to whatever something else might.

Exhaustive list was not important before, because the content of this directory did not go anywhere, so whatever Cabal produced (ever) would just be there. With this change - it started mattering.

It would be possible to describe movement of the most common directories in the changelog entry, but I'm not sure it's useful in the user's guide itself.

It is darn useful for somebody who's trying to migrate, e.g., dynamic build (100% of all Cabal build on my systems, if you care to know). I did not see where ~/.cabal/store is supposed to go.

You only have conflicts when you have identically named programs installed by different package managers, which seems to be a rare case.

It is pretty rare between, e.g., Cabal and Macports or Apt. It is practically 100% between your proposed Cabal and Stack.

cabal at least makes it easy to change the installation directory in the configuration file

Does it work for dynamically-linked executables (which have always been second-class citizens)? Where would the dependencies (dynamic/shared libraries) go, and will the loader (that load the binary in question to run) be able to locate them?

This concern was also brought up in the original issue: #680

I'm glad.

But in #680 you claimed that Cargo uses ~/.local/bin?! Nothing can be further from the truth. Download Rust (e.g., via rustup) and see for yourself that Cargo uses - you must've guessed it by now - ~/.cargo/bin. Actually, I think only Stack and Mono widely use ~/.local, though I seem to recall another package manager (not wide-spread at all, but still used by some) whose name eludes me at the moment.

[athas]

I did not see where ~/.cabal/store is supposed to go.

That might be worth mentioning explicitly. I'll make a PR that adds it.

Does it work for dynamically-linked executables (which have always been second-class citizens)? Where would the dependencies (dynamic/shared libraries) go, and will the loader (that load the binary in question to run) be able to locate them?

Where did they go previously? Since store paths are cryptographic hashes, I'm assuming dynamically linked executables embedded the paths, since there is no way the dynamic linker could find them otherwise. This should still work. My understanding is that the dir ~/.cabal/lib (now ~/.local/lib) was used only for "foreign libraries", i.e. Haskell libraries compiled to be callable from C. And while this is not a feature I ever used myself, I got the impression that users of this feature inevitably end up copying the .so files elsewhere anyway.

I can try building a dynamic binary and see if it still works.

[athas]

As expected, dynamically linked Haskell executables contain full store paths. They continue to work as well as they ever did.

[mouse07410]

Where would the dependencies (dynamic/shared libraries) go, and will the loader (that load the binary in question to run) be able to locate them?

Where did they go previously?

AFAIK, to the GHC version-related subdirectory of ~/.cabal/store.

My understanding is that the dir ~/.cabal/lib (now ~/.local/lib) was used only for "foreign libraries", i.e. Haskell libraries compiled to be callable from C. And while this is not a feature I ever used myself, I got the impression that users of this feature inevitably end up copying the .so files elsewhere anyway.

Well, I use it, and often do not copy the .so (or, in my case, .dylib) elsewhere. Sometimes I do. It really depends.

[mouse07410]

OK, this got merged - now, what's the answer to the question "where does ~/.cabal/store get migrated to"?

[athas]

$XDG_STATE_HOME/cabal/store

[jneira]

I know it is maybe too late (o is not?) but maybe it could be useful to make migrations smoother:

• in a minor o major release make it explicitly opt-in with a dedicated global flag: use-xdg-spec or whatever
  • setting it to false in new config files
  • document which are the changes, i.e. where will go each folder and possible caveats
• in the next major release set the global flag to true when writing new config files but honour existing ones flag value
  • and maybe warn about the fact? to encourage it's adoption?
  • maybe warn if you are using xdg explicitly but you still has some .cabal directory
• in the next major release (two after adding the feature) remove the flag handling from code, warn if it is set, and use xdg spec unconditionally

[athas]

in a minor o major release make it explicitly opt-in with a dedicated global flag: use-xdg-spec or whatever

This is a bit circular, since XDG behaviour changes the location of the config file. Also, people can already use XDG right now if they want, since all paths can be set in the config file (except for the location of the config file itself). In truth, all the XDG change really does is:

• Change the default location of the config file.
• Change the default directories when generating a config file (or equivalently, the default when the fields are missing from the config file).

A general cabal principle seems to be that all directories are supposed to be specified in the config file, and "using XDG" is mostly about selecting XDG directories by default when generating it. Only the location of the config file itself is (relatively) baked-in.

If you want, you can get XDG-equivalent behaviour with released cabal by setting an environment variable CABAL_CONFIG=$XDG_CONFIG_HOME/cabal/config and then setting the various directories in the config file (although they have to be constant, as cabal config files cannot use environment variables).

[jneira]

And agree an exhaustive list of where will go actual directories is essential to help in the migration.
Imo this migration mini guide should be included in the release notes but in a specific section at the very beginning to make it visible

[mouse07410]

Am I correct assuming that the old ~/.cabal location will always be supported?

[athas]

I can't predict the future, but I personally see no reason to remove it. It's useful in some cases, e.g. CI. I would say it is likely that at some point it will require you to set the CABAL_DIR environment variable to point to it (i.e. automatic backwards compatibility may not stick around forever).