'cabal new-build test:test-suite' doesn't force tests to be enabled.

[grayjay]

If there is a conflict involving the test dependencies, the solver disables the tests. I think that is because tests are only enabled with a preference in new-build. The build fails later with "cabal: Encountered missing dependencies: ...", unless the tests' dependencies are already required by another component.

name:          example
version:       1
cabal-version: >=1.2
build-type:    Simple

library

test-suite test-suite
  type:          exitcode-stdio-1.0
  main-is:       Tests.hs
  build-depends: unknown

$ cabal new-build test:test-suite 
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - example-1 (test:test-suite) (first run)
Configuring example-1...
cabal: Encountered missing dependencies:
unknown -any

Solver log with -v3:

Resolving dependencies...
targets: example
constraints: 
  base installed (non-upgradeable package)
  ghc-prim installed (non-upgradeable package)
  integer-gmp installed (non-upgradeable package)
  example ==1 (user target)
preferences: 
  example [TestStanzas,BenchStanzas]
  ADPfusion <0.4.0.0 || >0.4.0.0
  [...]
  zip-archive <0.2.3 || >0.2.3 && <0.2.3.1 || >0.2.3.1
strategy: PreferLatestForSelected
reorder goals: False
count conflicts: True
independent goals: False
avoid reinstalls: False
shadow packages: False
strong flags: False
max backjumps: 2000
[__0] trying: example-1 (user goal)
[__1] trying: example-1:*test
[__2] unknown package: unknown (dependency of example-1:*test)
[__2] fail (backjumping, conflict set: unknown, example-1:test)
[__1] trying: example-1:!test
[__2] done

[ezyang]

Thanks. @grayjay, do you know if the solver will only disable the test if there is NO solution that works with them enabled? If so, I think it's just a matter of giving a better error when a test target is specified.

If we do have to make the solver operate differently if a test is requested, we're in a bit more trouble, because then building a test target / not building a test target could lead to dep plan flip flopping. Not a good thing.

Re priority, this bug doesn't seem particularly urgent? Just a bad error message in an unlikely scenario.

[grayjay]

I think there was a solution with tests enabled when I first found this bug. I don't remember what I did, though. I just tried modifying the test case above, but then I discovered another bug that can prevent the solver from disabling tests. I'll create an issue for that later.

If testing is only a preference, the solver may disable tests when a solution with tests enabled exists. Do you think it would be better to enable them with a hard constraint in new-build? Then the user could specify --disable-tests if the tests can't be built, and the targets wouldn't affect the install plan.

[ezyang]

I think that's OK, but there may be unintended consequences I can't see.

[dcoutts]

Right, it's important that the target the user picks does not affect the solution. So we have two obvious UI choices:

• enable tests/benchmarks by default as a hard constraint, so failing if they cannot be enabled. Tell users they can try again with --disable-tests/--disable-benchmarks
• enable tests/benchmarks if possible, and don't fail if they cannot be enabled. Warn when they cannot be enabled. Tell users when they try to build disabled tests/benchmarks that they should use --enable-tests/--enable-benchmarks.

I think the second option is better. Much of the time you don't need to build the tests/benchmarks, so you don't need to notice or care if they cannot be easily built. Given our current solver error messages we cannot produce a good error message that says that the user can try again with tests/benchmarks disabled (ie we cannot currently determine if the solver failure is due to the test suites or something else).

What we're currently missing in option 2, is that we don't check and produce good error messages when a disabled testsuite/benchmark is requested, but this shouldn't be too hard to do.

[phadej]

@dcoutts, how about the projecs. At work I quite often have local patches to Hackage packages, and then my private code. I do not want these additional modified Hackage packages tests/benchmarks to affect my private build plan, because at some point I'll use a Hackage version.

In stack one could achieve this (kind of) by having

packages:
  - "."
  - location: github/
    extra-dep: true

As a consequence stack test doesn't run github tests either. I think it's quite common setup, so we should provide a way to specify whether tests/benchmarks are enabled/disabled by default per package in the cabal.project. Or do we already?

I.e. for my packages I build tests always, but for those Hackage dependencies not so often, and probably in isolation (something like cabal new-build github:tests --independent-plan would be cool), though there are in the project.

[ezyang]

@phadej I think if you add tests: False to the project descriptor that will have the desired effect. Of course you have to add this to every according project.

[dcoutts]

Right, either you can specify

package blah
  tests: False

for those local packages that you're patching, or perhaps alternatively we want to have control over which packages new-build considers to be "local" for the purposes of applying config & defaults. If you define those packages to be non-local then enabling tests for local packages (and other config given on the command line) would not apply to those packages.

[grayjay]

It looks like #4179 added an error message for the case where the user specifies a test target but the solver doesn't enable tests. I tried the original example with current master and got

$ cabal new-build test:test-suite 
cabal: Cannot build the test suite 'test-suite' because the solver did not
find a plan that included the test suites for example-1. It is probably worth
trying again with test suites explicitly enabled in the configuration in the
cabal.project{.local} file. This will ask the solver to find a plan with the
test suites available. It will either fail with an explanation or find a
different plan that uses different versions of some other packages. Use the
'--dry-run' flag to see package versions and check that you are happy with the
choices.

[andreasabel]

@grayjay wrote:

It looks like #4179 added an error message for the case where the user specifies a test target but the solver doesn't enable tests.

Yeah, and I find this message not very helpful. I'd rather see the usual output of cabal build from which I can deduce which constraints are not satisfiable.

In particular, I expected cabal build my-test-suite to try and build just this testsuite, giving me errors for just this testsuite. In my situation, there are two testsuites and the other one will definitely fail to build. But when I do cabal build my-test-suite --enable-tests, it tries to solve the constraints for all testsuites, which is bound to fail.

[Mikolaj]

Am I mistaken that on master we now do --enable-tests automatically for cabal test invocations? If so, this can probably be closed?

@andreasabel: thank you for your feedback. Feel free to try the new behaviour and add comments in newer tickets specifically about them. BTW, note that a lot of recent extra messages (and the recent --enable-tests change) are aimed at newcomers and we, old hands, just have to suffer through them. I think that's fair.

[ulysses4ever]

@Mikolaj no, this hasn't been done afaik. Still dealing with the fallout from cabal haddock enabling documentation 🥲

@andreasabel I think there's a separate ticket for per component solving and the vibe there: it's nowhere near.

[ulysses4ever]

#7883 and #5079 are for enabling tests automatically for cabal test. This ticket is a bit different though because it's about cabal build. It will be a bit tricker to implement.

[andreasabel]

@ulysses4ever : Thanks, indeed my concern was about cabal build my-test, and this ticket was sufficiently similar, so I didn't open a new one.

@Mikolaj: I am using cabal master whenever possible (so even this time).
(This is why I am reporting regressions all the time. ;-))

This PR would be great to have, but is seems to have stalled:

• [WIP] Only buildable tests #7829

[Mikolaj]

You are both right.

[grayjay]

I think that we should fix this issue with the same solution that was being discussed in the latest comments on #7883 (solving for tests by default, similarly to --enable-tests, but without building them). #7829 would need to be updated for the new design. I think that the set of targets specified on the command line shouldn't influence dependency solving, because it would lead to flip flopping build plans.