Add cabal freeze files and use them automatically #1561

hasufell · 2020-01-06T21:32:01Z

The freeze files are optional and shake will pick them up if they exist for the current requested ghc version.

alanz · 2020-01-06T22:02:43Z

Is this to guarantee reproducibility between the cabal and stack builds?

What is the process for updating them, from stack, and from cabal?

hasufell · 2020-01-06T22:06:06Z

Is this to guarantee reproducibility between the cabal and stack builds?

No. I don't care about the stack builds. Freeze files are meant for application authors to ensure users can build: https://www.haskell.org/cabal/users-guide/nix-local-build.html#cabal-v2-freeze

What is the process for updating them, from stack, and from cabal?

I have no idea how to use stack, this has nothing to do with stack. Also see https://www.haskell.org/cabal/users-guide/nix-local-build.html#cabal-v2-freeze

I manually added the with-compiler line to it after generating them, which I find useful.

fendor · 2020-01-06T23:01:54Z

Out of interest, did you test it somehow that it works as intended?

hasufell · 2020-01-06T23:02:17Z

Out of interest, did you test it somehow that it works as intended?

Yes, I built with ghc-8.6.5 and ghc-8.4.4.

alanz · 2020-01-06T23:06:07Z

This is probably a good idea

alanz · 2020-01-06T23:07:26Z

But I think we need some semi automatic way of updating them, without hand-editing afterward.
freeeze-update 8.6.5 or something like that.

fendor

Not sure if I like big generated files, but it does bring us one step closer to having a reproducible CI for cabal. It entails some more maintenance burden, though, but probably, only once a month?

hasufell · 2020-01-06T23:09:54Z

but probably, only once a month

Something like that.

But I think we need some semi automatic way of updating them, without hand-editing afterward.
freeeze-update 8.6.5 or something like that.

There's a ticket about it: haskell/cabal#4071

alanz · 2020-01-06T23:09:59Z

@fendor Having exactly reproducable builds with cabal is important, as it becomes a more popular delivery mechanism. And it is largely similar to what we are doing with stack, for the non-snapshot deps in the stack-XXX.yaml files.

So yes, some process that allows us to update, either when needed deps change (like hlint or the like), or as part of the monthly release cycle.

hasufell · 2020-01-06T23:12:17Z

Having exactly reproducable builds with cabal is important, as it becomes a more popular delivery mechanism.

There's one gotcha. To be 100% reproducible, we would need to freeze the hackage index as well (which can be done too). In some rare cases, hackage revisions (which are in-place updates to .cabal files) can retroactively break a build plan. This doesn't happen often, but can. I would omit this step though, unless it becomes a problem.

alanz · 2020-01-06T23:23:29Z

we would need to freeze the hackage index as well (which can be done too).

Would that not be enough? Which is a lot simpler

hasufell · 2020-01-06T23:35:53Z

Would that not be enough? Which is a lot simpler

How would that be enough? A hackage state has not just one version of a package.

alanz · 2020-01-06T23:41:01Z

How would that be enough? A hackage state has not just one version of a package.

It provides a frozen input to the solver, so should result in a frozen output. As I understand it.

hasufell · 2020-01-06T23:56:17Z

It provides a frozen input to the solver, so should result in a frozen output. As I understand it.

I think there may be more affecting the solver than just the hackage state (e.g. already installed packages, global config?, ...).

jneira · 2020-01-07T06:24:30Z

cabal.project.8.4.4

@@ -0,0 +1 @@
+cabal.project


Shouldn't that include the with-compiler = ghc-8.4.4?

Why? That makes it harder to maintain cabal.project and haskell/cabal#4071 suggests otherwise.

well as it is you could do a cabal --project-file cabal.project.8.4.4 and it would use the ghc on $PATH (ghc-8.6.5 f. e.) although you has ghc-8.4.4 in $PATH. Maybe doing that makes no sense but then this one could be removed, or is it needed for some reason?

EDIT: i guess it is needed to make cabal use the freeze version indirectly but the script could search for the .freeze directly and remove this one.

well as it is you could do a cabal --project-file cabal.project.8.4.4 and it would use the ghc on $PATH (ghc-8.6.5 f. e.)

cabal --project-file cabal.project.8.4.4 will use cabal.project.8.4.4.freeze unconditionally.

Yeah, i edited my comment

jneira · 2020-01-07T06:26:00Z

cabal.project.8.6.5

@@ -0,0 +1 @@
+cabal.project


Same here about with-compiler

jneira · 2020-01-07T06:28:03Z

There is no .project file for the rest of 8.4 and 8.5 minor versions, so it is not in sync with stack-*.yaml files, is it on purpose?

fendor · 2020-01-07T11:02:16Z

@jneira Stack and cabal are not in sync with these files. Rather we provide for cabal a mechanism to compile on any system with the same build-plan. It is in general decoupled of stack.
To have the same build plans between stack and cabal, we could also use: https://hackage.haskell.org/package/stack2cabal
It looks like with this project, it is trivial to have the same build-plan between stack and cabal.

jneira · 2020-01-07T11:21:44Z

I somewhat expect that would be a cabal.project.${ghcVersion} for each stack-${ghcVersion}.yaml file, to let cabal users build with the same ghc's.
What steps need to be done if a cabal user want to build with ghc-8.6.4 instead ghc-8.6.5, f.e.?

hasufell · 2020-01-07T12:32:03Z

There is no .project file for the rest of 8.4 and 8.5 minor versions, so it is not in sync with stack-*.yaml files, is it on purpose?

Cabal has nothing to do with stack. There is nothing to sync. freeze files are optional and only added for things that someone has tested.

What steps need to be done if a cabal user want to build with ghc-8.6.4

Nothing. Same as what is already in the README. It will simply default to using the original cabal.project without a freeze file. If you want to add freeze files for all ~10 compiler versions, please do so. That's not the scope of this PR.

jneira · 2020-01-07T12:51:37Z

Cabal has nothing to do with stack

Yeah i think it is clear enough 😉. I was talking about similar functionality using different tools.

That's not the scope of this PR.

Ok, i think using the last minor version will cover enough use cases. We could add the rest of them afterwards

install/src/Cabal.hs

code is correct as is

hasufell · 2020-01-07T13:23:47Z

@jneira hold on with the merge, I'll add freeze files for the other versions. I'm installing all ghc versions via ghcup.

I'll also add a script for regenerating them.

jneira · 2020-01-07T13:37:05Z

Well i think i am still used to stack.yaml files and the more similar cabal one maybe could be create a cabal.project.8.6.5 like:

packages:
         ./
         ./hie-plugin-api/

tests: true

index-state: 2020-01-02T03:40:04Z

with-compiler: ghc-8.6.5

index-state: 2020-01-02T03:40:04Z

package haskell-ide-engine
  test-show-details: direct

write-ghc-environment-files: never

-- Match the flag settings we use in stack builds
constraints: any.Cabal ==2.4.0.1,
             any.Diff ==0.4.0,
             any.HUnit ==1.6.0.0,
             any.ListLike ==4.6.3,
             any.QuickCheck ==2.13.2,
             QuickCheck +templatehaskell,
             any.SHA ==1.6.4.4,
             SHA -exe,
             any.StateVar ==1.2,
             any.Win32 ==2.6.1.0,
             -- rest of pinned deps from cabal.project.freeze for that ghc version 
             -- and hackage index state

But i suppose it doesnt follow cabal conventions. I still don fully like to have "dummy" (for me) cabal.project.8.6.5 that does not enforce the compiler version in any way (or am i missing something too here?). Note the use of index-stateto pin the hackage state.

hold on with the merge, I'll add freeze files for the other versions. I'm installing all ghc versions via ghcup.

ok, thanks for that

hasufell · 2020-01-07T13:55:58Z

Done, please check if you can run regen-freeze-files.sh.

lukel97

Nitpick: can we move these cabal.project.GHC_VER files into the install directory if they are to mainly be used by the install script?

hasufell · 2020-01-07T14:10:23Z

Nitpick: can we move these cabal.project.GHC_VER files into the install directory if they are to mainly be used by the install script?

They are not mainly used for the install script.

lukel97 · 2020-01-07T14:17:52Z

Nitpick: can we move these cabal.project.GHC_VER files into the install directory if they are to mainly be used by the install script?

They are not mainly used for the install script.

Is it to be mainly used by cabal --project-file cabal.project.8.x.x then? I.e. if the user decides to install hie via plain cabal, and not via shake?

hasufell · 2020-01-07T14:20:12Z

Is it to be mainly used by cabal --project-file cabal.project.8.x.x then? I.e. if the user decides to install hie via plain cabal, and not via shake?

As already explained: freeze files correspond to the project files. The project file lives at the top of the project hierarchy. The project and freeze files are used to build hie (with or without the shake build system).

lukel97 · 2020-01-07T14:33:20Z

Is it to be mainly used by cabal --project-file cabal.project.8.x.x then? I.e. if the user decides to install hie via plain cabal, and not via shake?

As already explained: freeze files correspond to the project files. The project file lives at the top of the project hierarchy. The project and freeze files are used to build hie (with or without the shake build system).

My wording was unclear: is the intention that people will run cabal --project-file cabal.project.8.x.x install to install and build hie? Because as @jneira mentioned they will need to also specify --with-compiler=ghc-8.x.x. And I'm not 100% convinced that the method of installing multiple versions of hie via cabal by hand is that common

hasufell · 2020-01-07T14:39:13Z

My wording was unclear: is the intention that people will run cabal --project-file cabal.project.8.x.x install to install and build hie?

Again: they can do that or use shake.

Because as @jneira mentioned they will need to also specify --with-compiler=ghc-8.x.x.

No, they don't have to specifiy that.

lukel97 · 2020-01-07T14:51:11Z

Because as @jneira mentioned they will need to also specify --with-compiler=ghc-8.x.x.

No, they don't have to specifiy that.

I see, the freeze file seems to contain the compiler id part. I'm getting sold on this idea now, but I still think we should put them into a separate cabal-projects directory, since the top level is getting pretty badly cluttered. cabal build --with-project=cabal-projects/cabal.project.8.6.5 still seems to pick up the freeze file for me.

As for the 8.6.5 freeze file, I'm not able to build it:

$ cabal build --project-file cabal.project.8.6.5
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: haskell-ide-engine-1.0.0.0 (user goal)
[__1] next goal: hoogle (dependency of haskell-ide-engine)
[__1] rejecting: hoogle-5.0.17.13, hoogle-5.0.17.12 (constraint from project
config TODO requires ==5.0.17.11)
[__1] trying: hoogle-5.0.17.11
[__2] trying: warp-tls-3.2.9 (dependency of hoogle)
[__3] next goal: tls-session-manager (dependency of warp-tls)
[__3] rejecting: tls-session-manager-0.0.3, tls-session-manager-0.0.2.1,
tls-session-manager-0.0.2.0, tls-session-manager-0.0.1.0,
tls-session-manager-0.0.0.2, tls-session-manager-0.0.0.1,
tls-session-manager-0.0.0.0 (constraint from project config TODO requires
==0.0.4)
[__3] fail (backjumping, conflict set: tls-session-manager, warp-tls)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: hoogle, warp-tls, tls-session-manager,
haskell-ide-engine
Try running with --minimize-conflict-set to improve the error message.

hasufell · 2020-01-07T14:57:50Z

into a separate cabal-projects directory

That's not how project files are to be used. And it might break tools (which search upwards for project files). They are supposed to be in top-level.

As for the 8.6.5 freeze file, I'm not able to build it:

Yes, because you didn't run cabal update.

lukel97 · 2020-01-07T15:03:10Z

That's not how project files are to be used. And it might break tools (which search upwards for project files). They are supposed to be in top-level.

These project files won't be picked up by any tools since they don't match "*.project", at least not for the hie-vscode-languageserver extensions or any of the default project markers in the README anyway.

Yes, because you didn't run cabal update.

Might be worthwhile looking into adding index-state: into the cabal.project files. And then we basically have stack snapshots for cabal!

hasufell · 2020-01-07T15:07:05Z

These project files won't be picked up by any tools since they don't match "*.project", at least not for the hie-vscode-languageserver extensions or any of the default project markers in the README anyway.

I don't understand what that means.

Might be worthwhile looking into adding index-state: into the cabal.project files. And then we basically have stack snapshots for cabal!

As explained, I don't think that is necessary right now. It's kind of a heavy shoot, imo.

jneira · 2020-01-07T15:28:57Z

I am not gonna ask for more changes, but i want to note for future follow ups, as a reminder, that the pinned deps (and the script to generate them) are os dependent (correct me if i am wrong)
So it will be needed to add a version, at least, for windows, change install.hs to take them in account and translate the script to an os independent lang.

hasufell · 2020-01-07T15:38:30Z

So it will be needed to add a version, at least, for windows, change install.hs to take them in account and translate the script to a os independent lang.

Not sure that is necessary. This script doesn't have to be run by users at all and only ~once a month by one developer. Surely there's someone running linux.

I am not gonna ask for more changes, but i want to note for future follow ups, as a reminder, that the pinned deps (and the script to generate them) are os dependent

The worst case is that windows-specific deps don't make it into the freeze file. Are there any deps that only exist on windows? If not, then there's no issue.

jneira · 2020-01-07T16:07:04Z

Mmm i can do a diff but as we can see in the example i posted above it has any.win32==2 6.1.0 (and linux will have unix). I bet there will be more cases of libs usables for both os but only present in one of them.
I will test the pr freeze files in windows to make sure they work.

We can accept the possible risk of failing builds in windows due to unpinned deps but i would prefer to have parity between os's.

If we want those os specific files we'll need generate them in that os and i would prefer again to not have to do it manually.

hasufell · 2020-01-07T18:30:38Z

@alanz cabal developers wrt only using index file:

maerwald: phadej: do you know the exact difference between using a freeze file and pinning the hackage state? If the build plan only depends on 1. the .cabal file and 2. the hackage state, then that should be enough for reproducible output, right?
maerwald: But my gut feeling tells me that might not be the case
maerwald: (e.g. already installed packages might change the build plan?)
jojoz[m]: oof it broke
phadej: pinning index-state and cabal-install version should produce same install plan yes, except if there's some pkg-config etc. global change which may flip some automatic flags
phadej: already installed packages don't affect v2-build solver at all
phadej: (that's the point)
maerwald: cabal options like max-backjumps and whatnot don't affect the build plan?
phadej: max-backjumps don't, some other might (like reorder-goals)
phadej: but not in the case when the resulting plan is "all the latest (up to index-state)"
maerwald: pinning index state seems easier when you want to support both windows and unix
phadej: it depends how much freedom in movement there is (for apps I'd recommend to support only one major version of dependencies anyway, so most likely solver will find always the same plan)
phadej: problematic cases are "should I pick newer vector or newer hashable, because of the dependencies I can pick only either, but not both"
phadej: and those build-plans aren't "comparable" (you cannot say which one is better, objectively)

I've never used this approach, so I cannot comment on how smooth it is. I guess we could give that a try instead. If we are lucky, one index state will work for all ghc versions even. But I'll have to test that. We would also need to pin cabal-install somehow.

However, the documentation of index-state still suggests to use a freeze file: https://www.haskell.org/cabal/users-guide/nix-local-build.html#cfg-field-index-state

This allows to change the source package index state the solver uses to compute install-plans. This is particularly useful in combination with freeze-files in order to also freeze the state the package index was in at the time the install-plan was frozen.

Another downside of using index-state only would be that you don't really know if you broke the solver, once you changed the .cabal file (especially thinking upper bounds). But that could be a job for CI.

hasufell · 2020-01-07T22:41:00Z

update-index-state.sh is sorta tricky, but it is only there to aid development. It isn't strictly necessary.

This is kinda heavy, but it should work well on 32bit and 64bit arches. The alternative was to copy-paste a lot of cabal-install code that is not exposed. There seems to be no value in that for a development script. The alternative to this script is simply running 'cabal v2-update' twice and then copy-pasting the output manually to cabal.project.

This reverts commit 93ecb1e.

This reverts commit 8e61c59.

jneira · 2020-01-08T06:38:50Z

To be honest , it seems to me that freeze files was not designed for being distributed but to be automatically generated by each developer and git ignored (like cabal.project.local).

Another downside of using index-state only would be that you don't really know if you broke the solver, once you changed the .cabal file (especially thinking upper bounds). But that could be a job for CI.

That limits somewhat the solution (CI checks already builds although you don't pin the index state) but to be fair it is still a improvement.

Btw, is it possible that, in a specific hackage index state, the builds work for an os but not for others for any ghc version?
Well i suppose CI (again) can check that and we'll have to select another one (manually?) in that case. I hope that it will not be very common.

hasufell · 2020-01-08T12:43:54Z

To be honest , it seems to me that freeze files was not designed for being distributed but to be automatically generated by each developer and git ignored

The documentation says the opposite:

For end-user executables, it is recommended that you distribute the cabal.project.freeze file in your source repository so that all users see a consistent set of dependencies.

Freeze files are meant to be distributed. What you mean is cabal.project.local, for local development.

fendor approved these changes Jan 6, 2020

View reviewed changes

fendor requested review from jneira and alanz January 6, 2020 23:08

jneira suggested changes Jan 7, 2020

View reviewed changes

hasufell requested a review from jneira January 7, 2020 12:43

jneira previously requested changes Jan 7, 2020

View reviewed changes

install/src/Cabal.hs Outdated Show resolved Hide resolved

jneira approved these changes Jan 7, 2020

View reviewed changes

lukel97 reviewed Jan 7, 2020

View reviewed changes

hasufell added 2 commits January 7, 2020 15:10

Add cabal freeze files and use them automatically

8e61c59

Add script to regenerate freeze files

93ecb1e

Pin index state

b202c8b

hasufell requested review from fendor and jneira January 7, 2020 22:34

hasufell added 3 commits January 7, 2020 23:55

Revert "Add script to regenerate freeze files"

8164e43

This reverts commit 93ecb1e.

Revert "Add cabal freeze files and use them automatically"

f2881f2

This reverts commit 8e61c59.

fendor approved these changes Jan 8, 2020

View reviewed changes

jneira approved these changes Jan 9, 2020

View reviewed changes

jneira merged commit 92ac896 into haskell:master Jan 10, 2020

hasufell mentioned this pull request Mar 8, 2021

Drop hackage-index from cabal.project haskell/haskell-language-server#1496

Closed

jneira mentioned this pull request Mar 23, 2022

Strategies for managing Cabal freeze files with platform-specific dependencies haskell/cabal#8059

Closed

Add cabal freeze files and use them automatically #1561

Add cabal freeze files and use them automatically #1561

Conversation

hasufell commented Jan 6, 2020 • edited Loading

alanz commented Jan 6, 2020

hasufell commented Jan 6, 2020 • edited Loading

fendor commented Jan 6, 2020

hasufell commented Jan 6, 2020

alanz commented Jan 6, 2020

alanz commented Jan 6, 2020

fendor left a comment

Choose a reason for hiding this comment

hasufell commented Jan 6, 2020

alanz commented Jan 6, 2020

hasufell commented Jan 6, 2020

alanz commented Jan 6, 2020

hasufell commented Jan 6, 2020

alanz commented Jan 6, 2020

hasufell commented Jan 6, 2020

jneira Jan 7, 2020

Choose a reason for hiding this comment

hasufell Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

jneira Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

hasufell Jan 7, 2020

Choose a reason for hiding this comment

jneira Jan 7, 2020

Choose a reason for hiding this comment

jneira Jan 7, 2020

Choose a reason for hiding this comment

jneira commented Jan 7, 2020

fendor commented Jan 7, 2020

jneira commented Jan 7, 2020

hasufell commented Jan 7, 2020 • edited Loading

jneira commented Jan 7, 2020

hasufell commented Jan 7, 2020 • edited Loading

jneira commented Jan 7, 2020 • edited Loading

hasufell commented Jan 7, 2020

lukel97 left a comment

Choose a reason for hiding this comment

hasufell commented Jan 7, 2020

lukel97 commented Jan 7, 2020

hasufell commented Jan 7, 2020 • edited Loading

lukel97 commented Jan 7, 2020

hasufell commented Jan 7, 2020

lukel97 commented Jan 7, 2020

hasufell commented Jan 7, 2020

lukel97 commented Jan 7, 2020

hasufell commented Jan 7, 2020

jneira commented Jan 7, 2020 • edited Loading

hasufell commented Jan 7, 2020

jneira commented Jan 7, 2020 • edited Loading

hasufell commented Jan 7, 2020

hasufell commented Jan 7, 2020

jneira commented Jan 8, 2020 • edited Loading

hasufell commented Jan 8, 2020 • edited Loading

hasufell commented Jan 6, 2020 •

edited

Loading

hasufell commented Jan 6, 2020 •

edited

Loading

hasufell Jan 7, 2020 •

edited

Loading

jneira Jan 7, 2020 •

edited

Loading

hasufell commented Jan 7, 2020 •

edited

Loading

hasufell commented Jan 7, 2020 •

edited

Loading

jneira commented Jan 7, 2020 •

edited

Loading

hasufell commented Jan 7, 2020 •

edited

Loading

jneira commented Jan 7, 2020 •

edited

Loading

jneira commented Jan 7, 2020 •

edited

Loading

jneira commented Jan 8, 2020 •

edited

Loading

hasufell commented Jan 8, 2020 •

edited

Loading