foliage caching issues

[michaelpj]

I had some issues where foliage was broken and I had to delete the cache to continue. I suspect that means we're not invalidating things properly sometimes.

If we could fix this, then we could e.g. cache the cache directory in GHA, saving some CI time. I hypothesize that this would save quite a bit, because the builds of the main/tip repositories for PRs in CHaP accidentally share the cache, and the second build is much faster. That makes sense, since presumably it lets it skip downloading all the stuff that's the same.

[andreabedini]

Yes, this has appened to me too and it was the ause of this failure
https://github.com/input-output-hk/cardano-haskell-packages/actions/runs/4380948045/jobs/7669285627#step:8:2190

I could reproduce it reliably doing foliage build; rm -rf _repo; foliage build. This change seemed to fix the issue but I am worried I am misunderstanding how shake works. I started foliage by using standard "rule based" shake, and later switched to something that looks more like forward mode (despite not relying on fs-tracing). If you have time, I'd appreciate some feedback on this, maybe I am holding shake wrong :)

[michaelpj]

I also don't really understand how it works tbh. I did wonder whether maybe the directories we use within the cache directory need to be properly keyed by all the information in the input (perhaps a hash), so we don't reuse stuff we shouldn't?

[michaelpj]

If we could fix this, then we could e.g. cache the cache directory in GHA, saving some CI time.

We already do this 🤦

the second build is much faster

That's actually because the first one has to download all the Nix deps 🤦

[andreabedini]

I am going to keep this issue to track caching issues. Comment here if you have any new issue with caching.