solver: Interaction of independent-goals and constraints is not clear #9466

mpickering · 2023-11-24T10:10:58Z

It would be worthwhile to improve the documentation about how independent-goals is supposed to interact with user specified constraints in a cabal.project or given on the command line.

For example if I have a package A which depends on B, if I give the solver the goal A and enable --independent-goals should a constraint --constraint= B == 2 apply inside the independent goal of A?

Reasons to consider:

The constraint does not apply inside other qualified contexts (for example, setup stanzas), so why does it apply inside a qualified goal.
If you want a constraint to apply within ALL goals then you can already write any.A rather than simply A.
The goal of A is supposed to be solved "indepedently" of the rest of the goals/context, this constraint is applied at a global rather than per-goal level.

Therefore it seems like a bug to me that the constraint affects the solver when deciding which version of B to pick when solving the independent goal for A.

cc @grayjay

This situation is witnessed by this test, the result is A == 1, B == 2.

 testIndepGoals8 :: String -> SolverTest                                         
 testIndepGoals8 name =                                                          
   constraints [ExVersionConstraint (scopeTopLevel "B") (V.thisVersion (V.mkVersion [2,0,0]))]  $
     independentGoals $                                                          
         mkTest db name ["A"] $                                                  
           solverSuccess [("A", 1), ("B", 1)]                                    
   where                                                                         
     db :: ExampleDb                                                             
     db =                                                                        
       [ Right $ exAv "A" 1 [ExAny "B"]                                          
       , Right $ exAv "B" 1 []                                                   
       , Right $ exAv "B" 2 []                                                   
       , Right $ exAv "B" 3 []                                                   
       ]

The text was updated successfully, but these errors were encountered:

These tests check how constraints interact with the --independent-goals flag.

grayjay · 2023-11-26T04:10:49Z

I think that the issue is that the independent goals feature was never designed to be used for anything except testing qualified goals or possibly acting as a workaround when another issue prevents cabal from finding a solution. I don't think that the feature is complete. Qualified constraints also isn't complete. #3502 tracks the remaining work, including constraints for build tool dependencies and qualified constraints in freeze files. If we wanted to fully support independent goals, I think we would need to allow qualifying constraints with the name of the target.

What is your use case for independent goals? Are you planning to build something on top of the feature?

mpickering · 2023-11-27T10:49:53Z

@grayjay I am implementing #4035 and perhaps also an extension which generalises the base shim logic.

There are some tests which fail after I have performed some refactoring of qualified goals, so I need to know what the specification is for this feature in order to know how to fix them.

grayjay · 2023-12-02T01:45:11Z

Okay, I think that the part of independent goals that is implemented is well defined; it just hasn't been made into a useful feature or added to the documentation yet. It looks like qualified constraints for independent goals also hasn't been implemented yet.

#3220 has the best description of independent goals that I have found. I believe that the feature just gives each build target a unique namespace (part of the PackagePath) in the dependency solver, and that namespace is inherited by all dependencies, including setup and build tool dependencies. Ignoring the rules of inheritance, PackagePaths with different namespaces are treated similarly to PackagePaths with different qualifiers.

I found an example of a use case for independent goals, in case that helps: #4295

Is there anything that I can read about the implementation of private dependencies? I'm wondering how it would affect the dependency solver. I'm especially interested in the effects on performance, since qualified goals have a big impact on solver run time, as in #6309.

mpickering · 2023-12-04T12:50:21Z

@grayjay Back to the original question:

For example if I have a package A which depends on B, if I give the solver the goal A and enable --independent-goals should a constraint --constraint= B == 2 apply inside the independent goal of A?

It seems you are suggesting that it shouldn't (which is the conclusion I came to).

These tests check how constraints interact with the --independent-goals flag.

@alt-romes

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita (@alt-romes)

@alt-romes

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita (@alt-romes)

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita <[email protected]>

These tests check how constraints interact with the --independent-goals flag.

testsuite: Add some unit tests for #9466 #9467

These tests check how constraints interact with the --independent-goals flag.

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita <[email protected]>

These tests check how constraints interact with the --independent-goals flag.

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita <[email protected]>

These tests check how constraints interact with the --independent-goals flag. (cherry picked from commit b169cd4)

* testsuite: Add tests for #9467 (base shim, setup qualifier interaction) This adds two tests for issue #9467 (cherry picked from commit 3851043) * testsuite: Add two tests for independent goals (#9466) These tests check how constraints interact with the --independent-goals flag. (cherry picked from commit b169cd4) * testsuite: Add some tests for setup component scope interacts with stanza flags These tests check how the setup qualified scope interacts with the stanza flags (specified on the top-level and with the any qualifier) (cherry picked from commit 573c15d) --------- Co-authored-by: Matthew Pickering <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

This commit introduces so-called "private dependencies". High-level Overview ~~~~~~~~~~~~~~~~~~~ Since its inception, Cabal has enforced the restriction that a library must only link against one version of each package it depends on. This ensures that all of the dependencies in the build plan work together. In your application you use different libraries together, so it’s of paramount importance that they all agree on what `Text` means or what a `ByteString` is. However, sometimes it’s desirable to allow multiple versions of the same library into a build plan. In this case, it’s desirable to allow a library author to specify a private dependency with the promise that its existence will not leak from the interface of the library which uses it. The two main use cases of private dependencies are: - Writing benchmarks and testsuites for your library which test new versions of your library against old versions. - Writing libraries which can communicate with processes built against a range of different library versions (such as cabal-install calling ./Setup). A user specifies a private dependency in their cabal file using `private-build-depends`. The specification starts with the name of the private dependency scope and then contains a list of normal dependency specifications which dictates what is included in that private scope: private-build-depends: TEXT1 with (text == 1.2.*), TEXT2 with (text == 2.*) Each private scope is then solved independently of all other scopes. In this example the TEXT1 scope can choose a version of text in the 1.2.x range and the TEXT2 scope can choose a version of text in the 2.* range. Private scopes do not apply transitively, so the dependencies of text will be solved in the normal top-level scope. If your program contains a value of type Bool, that comes from the base package, which text depends on, because the scopes are not applied transitively the same Bool value can be passed to functions from the TEXT1 scope and TEXT2 scope. Dependencies introduced privately can be imported into modules in the project by prefixing the name of the private scope to an exposed module name. import qualified TEXT1.Data.Text as T1 import qualified TEXT2.Data.Text as T2 Closure of Private Scopes ~~~~~~~~~~~~~~~~~~~~~~~~~ Private dependency scopes can contain multiple packages. Packages in the same scope are solved together. For example, if two packages are tightly coupled and you need to use compatible versions with each other, then you can list them in the same private scope. Such packages will then be solved together, but independently of other packages. Private scopes must be closed. A scope is closed if, whenever we have a dependency chain P1 -> Q -> P2, in which P1 and P2 are both in a given private scope S, then Q also belongs to the private scope S. The solver checks this property, but doesn’t implicitly add packages into a private scope. Implementation ~~~~~~~~~~~~~~ To implement private dependencies we changed * Cabal-syntax to introduce the new `private-build-depends: ALIAS (packages, in, private, scope)` syntax. See the new type `Dependencies` and changes in `Distribution.Types.Dependency`. * cabal-install-solver now considers both public and private dependencies of a given package (see e.g. `solverPkgLibDeps`), has a new constructor `PrivateScope` in `ConstraintScope` for goals in a private scope, and there's a new `Qualifier` for packages introduced in private scopes (see also [Namespace vs Qualifier refactor] below), to solve them separately from packages introduced by `build-depends`. * cabal-install-solver needs to check that the private-scope closure property holds (the closure of the packages in a private scope is in the private scope) (see `Distribution.Solver.Modular.PrivateScopeClosure`). We check that the closure holds by looking at the reverse dependency map while traversing down the tree, at every node: For every package in a private scope, traverse up the reverse dependency map until a package in the same private scope is found. If one exists, and if along the way up any package was not in the same private scope as the packages in the two ends, we fail. * cabal-install understands plans with private dependencies and has a new `UserQualifier` to support constrainting packages in private scopes using the `--constraint` flag. Example: `--constraint=private.pkg-a.TEXT01:text == 1.2.*` * Cabal the library uses the ghc module-renaming mechanism (also used by Backpack) to rename modules from the packages in a private scope to prefix them with the private scope alias. It also ensures `cabal check` fails if there exist the package has private dependencies, as it is currently an experimental feature which we don't necessarily want to allow in hackage yet -- e.g. how will haddock render private dependencies? Namespace vs Qualifier refactor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We also refactored the `Namespace` vs `Qualifier` types in the solver, clarifying their interaction such that: * A package goal with an indepedent namespace is fully solved indepently from other namespaces, i.e. all the dependency goals introduced by a goal in a given namespace are also solved in that namespace. * In contrast, a package goal with a qualifier is shallow-solved separately from other goals in the same namespace. The dependency goals introduced by it will be solved unqualified (`QualTopLevel`) in that namespace. For example, goal `pkg-a == 0.1` in `DefaultNamespace+QualTopLevel`, and goal `pkg-a == 0.2` in the same namespace but with `QualAlias A2 ...` can be solved together and yield a different version of pkg-a for each of the goals, however, the dependencies of both will be solved together -- if they both dependend on `base`, we'd have to find a single solution. If `pkg-a == 0.2` was in an `Independent` namespace, we could still solve the two goals with two versions of `pkg-a`, but we could also pick different versions for all the subdependencies of `pkg-a == 0.2`. Besides Namespace vs Qualifier being a welcome refactor that facilitates implementing private dependencies, it also fixes haskell#9466 and helps with haskell#9467. --- Co-authored-by: Rodrigo Mesquita <[email protected]>

mpickering added the type: user-question label Nov 24, 2023

Mikolaj added the documentation label Nov 24, 2023

mpickering added a commit to mpickering/cabal that referenced this issue Nov 24, 2023

testsuite: Add two tests for independent goals (haskell#9466)

58452a1

These tests check how constraints interact with the --independent-goals flag.

mpickering mentioned this issue Nov 24, 2023

testsuite: Add some unit tests for #9466 #9467 #9468

Merged

mpickering added a commit to mpickering/cabal that referenced this issue Nov 24, 2023

testsuite: Add two tests for independent goals (haskell#9466)

71cb443

These tests check how constraints interact with the --independent-goals flag.

grayjay added the cabal-install: solver label Nov 24, 2023

alt-romes pushed a commit to mpickering/cabal that referenced this issue Mar 11, 2024

testsuite: Add two tests for independent goals (haskell#9466)

e34eaf1

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Mar 11, 2024

testsuite: Add two tests for independent goals (haskell#9466)

d53cf61

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Mar 14, 2024

testsuite: Add two tests for independent goals (haskell#9466)

e64097c

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Apr 3, 2024

testsuite: Add two tests for independent goals (haskell#9466)

03739d1

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Apr 3, 2024

testsuite: Add two tests for independent goals (haskell#9466)

77acec9

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Apr 4, 2024

testsuite: Add two tests for independent goals (haskell#9466)

3e14f1d

These tests check how constraints interact with the --independent-goals flag.

alt-romes pushed a commit to mpickering/cabal that referenced this issue Apr 5, 2024

testsuite: Add two tests for independent goals (haskell#9466)

c827a54

These tests check how constraints interact with the --independent-goals flag.

Mikolaj pushed a commit to mpickering/cabal that referenced this issue Apr 7, 2024

testsuite: Add two tests for independent goals (haskell#9466)

b169cd4

These tests check how constraints interact with the --independent-goals flag.

mergify bot added a commit that referenced this issue Apr 7, 2024

Merge pull request #9468 from mpickering/wip/unit-tests

cd9b1fd

testsuite: Add some unit tests for #9466 #9467

alt-romes pushed a commit to mpickering/cabal that referenced this issue Apr 12, 2024

testsuite: Add two tests for independent goals (haskell#9466)

7b4527c

These tests check how constraints interact with the --independent-goals flag.

erikd pushed a commit to erikd/cabal that referenced this issue Apr 22, 2024

testsuite: Add two tests for independent goals (haskell#9466)

2ed3630

These tests check how constraints interact with the --independent-goals flag.

mergify bot pushed a commit that referenced this issue May 16, 2024

testsuite: Add two tests for independent goals (#9466)

63dc94f

These tests check how constraints interact with the --independent-goals flag. (cherry picked from commit b169cd4)

mergify bot mentioned this issue May 16, 2024

testsuite: Add some unit tests for #9466 #9467 (backport #9468) #10020

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

solver: Interaction of independent-goals and constraints is not clear #9466

solver: Interaction of independent-goals and constraints is not clear #9466

mpickering commented Nov 24, 2023

grayjay commented Nov 26, 2023

mpickering commented Nov 27, 2023

grayjay commented Dec 2, 2023

mpickering commented Dec 4, 2023

solver: Interaction of independent-goals and constraints is not clear #9466

solver: Interaction of independent-goals and constraints is not clear #9466

Comments

mpickering commented Nov 24, 2023

grayjay commented Nov 26, 2023

mpickering commented Nov 27, 2023

grayjay commented Dec 2, 2023

mpickering commented Dec 4, 2023