Cabal new-build generating insanely long and redundant build PATHS on Windows.

[erikd]

I'm trying to https://github.com/input-output-hk/cardano-sl on Windows in AppVeyor. Unfortunately Cardano SL is a large project and using Stack we have been continuously bumping up against the Windows path length limitations.

Trying cabal new-build as an alternative to stack we run into the Windows path length issue in a different place. From the AppVeyor output, it seems cabal new-build is installing stuff in:

C:\Users\appveyor\AppData\Roaming\cabal\store\ghc-8.0.2\incoming\new-1824\
  Users\appveyor\AppData\Roaming\cabal\store\ghc-8.0.2\

a path with one hell of a lot of redundancy. That path should at least be shortened to something like:

C:\Users\appveyor\AppData\Roaming\cabal\store\incoming\new-1824\ghc-8.0.2\

[harpocrates]

Does --store-dir not fix this for you? You should be able to set that to something super short... I was running into this sort of problem on AppVeyor and passing --store-dir=c:\cabal-store ended up being my workaround.

[gbaz]

But even if there's a workaround, it still seems like a genuine bug that the path is duplicated, no?

[Mistuke]

Yeah, I have a PR for that but couldn't figure out why it caused a test to fail on linux and I forgot about it... #4978

Also long paths for Haskell programs are fixed in GHC 8.6, so a cabal-install compiled with 8.6 should work without any workarounds.

[angerman]

Long paths may be fixed. The response file fix for library search paths, which what @erikd s underlying issue is, won’t be solved in time for 8.6, I believe?

[Mistuke]

It depends on when @bgamari wants patches by. I'm trying figured out the build issues.

…

On Mon, Jun 18, 2018, 00:29 Moritz Angermann ***@***.***> wrote: Long paths may be fixed. The response file fix for library search paths, which what @erikd <https://github.com/erikd> s underlying issue is, won’t be solved in time for 8.6, I believe? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABH3KUh7ePARmOn2VXr42DhQ3FgR8wVNks5t9uZ1gaJpZM4Uj5ZE> .

[Mistuke]

Also how is the response files thing Erik'd underlying issue? This ticket is purely about backpack and new-build making a path too long for it to write to with the current platform APIs it's using. It has nothing to do with the underlying compiler and linker because it doesn't get that far.

[angerman]

You are right that it’s not explicitly mentioned. I believe this came up when we tried to swap cabal in for stack, which kept failing on windows. The exceptionally long paths just made the issue worse.

[AndreasPK]

Also long paths for Haskell programs are fixed in GHC 8.6, so a cabal-install compiled with 8.6 should work without any workarounds.

I still get long path related errors with cabal new-build using GHC head so I'm not sure if this is completely fixed.

[Mistuke]

Such as?

…

On Fri, Jul 27, 2018, 09:04 AndreasPK ***@***.***> wrote: Also long paths for Haskell programs are fixed in GHC 8.6, so a cabal-install compiled with 8.6 should work without any workarounds. I still get long path related errors with cabal new-build using GHC head so I'm not sure if this is completely fixed. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABH3KQ7A_GmHQfyLObq3BnuiczL_mcl1ks5uKsmSgaJpZM4Uj5ZE> .

[AndreasPK]

Moving the whole folder into a shallower directory works.

Although it's possible that this is caused by the library issue discussed above?

Either way I still see errors with cabal using 8.6(HEAD) and long paths.

Resolving dependencies...
Build profile: -w ghc-8.7.20180718 -O1
In order, the following will be built (use -v for more details):
 - aeson-benchmarks-0 (exe:aeson-benchmark-aeson-encode) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-auto-compare) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-compare) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-compare-with-json) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-foldable) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-map) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-micro) (first run)
 - aeson-benchmarks-0 (exe:aeson-benchmark-typed) (first run)
Configuring executable 'aeson-benchmark-foldable' for aeson-benchmarks-0..
Preprocessing executable 'aeson-benchmark-aeson-encode' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-aeson-encode' for aeson-benchmarks-0..
Preprocessing executable 'aeson-benchmark-auto-compare' for aeson-benchmarks-0..
Preprocessing executable 'aeson-benchmark-compare-with-json' for aeson-benchmarks-0..
Preprocessing executable 'aeson-benchmark-compare' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-compare-with-json' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-auto-compare' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-compare' for aeson-benchmarks-0..
[1 of 1] Compiling Main             ( AesonEncode.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-aeson-encode\build\aeson-benchmark-aeson-encode\aeson-benchmark-aeson-encode-tmp\Main.o )
[1 of 1] Compiling Main             ( CompareWithJSON.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-compare-with-json\build\aeson-benchmark-compare-with-json\aeson-benchmark-compare-with-json-tmp\Main.o )
[ 1 of 10] Compiling Options          ( Options.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-auto-compare\build\aeson-benchmark-auto-compare\aeson-benchmark-auto-compare-tmp\Options.o )
Assembler messages:
Fatal error: can't create C:\\ghc\\msys64\\home\\Andi\\bench_script\\lib_bench\\aeson\\aeson_allCalls\\dist-vanilla\\build\\x86_64-windows\\ghc-8.7.20180718\\aeson-benchmarks-0\\x\\aeson-benchmark-auto-compare\\build\\aeson-benchmark-auto-compare\\aeson-benchmark-auto-compare-tmp\\Options.o: No such file or directory
[2 of 7] Compiling Compare.JsonBuilder ( Compare\JsonBuilder.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-compare\build\aeson-benchmark-compare\aeson-benchmark-compare-tmp\Compare\JsonBuilder.o )
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)
Configuring executable 'aeson-benchmark-map' for aeson-benchmarks-0..
Warning: 'c-sources: ../cbits/unescape_string.c' is a relative path outside of
the source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ..' is a relative path outside of the source tree.
This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../ffi' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../pure' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../attoparsec-iso8601' is a relative path outside of
the source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'include-dirs: ../include' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Assembler messages:
Fatal error: can't create C:\\ghc\\msys64\\home\\Andi\\bench_script\\lib_bench\\aeson\\aeson_allCalls\\dist-vanilla\\build\\x86_64-windows\\ghc-8.7.20180718\\aeson-benchmarks-0\\x\\aeson-benchmark-aeson-encode\\build\\aeson-benchmark-aeson-encode\\aeson-benchmark-aeson-encode-tmp\\Main.o: No such file or directory
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)
Assembler messages:
Fatal error: can't create C:\\ghc\\msys64\\home\\Andi\\bench_script\\lib_bench\\aeson\\aeson_allCalls\\dist-vanilla\\build\\x86_64-windows\\ghc-8.7.20180718\\aeson-benchmarks-0\\x\\aeson-benchmark-compare\\build\\aeson-benchmark-compare\\aeson-benchmark-compare-tmp\\Compare\\JsonBuilder.o: No such file or directory
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)
Assembler messages:
Fatal error: can't create C:\\ghc\\msys64\\home\\Andi\\bench_script\\lib_bench\\aeson\\aeson_allCalls\\dist-vanilla\\build\\x86_64-windows\\ghc-8.7.20180718\\aeson-benchmarks-0\\x\\aeson-benchmark-compare-with-json\\build\\aeson-benchmark-compare-with-json\\aeson-benchmark-compare-with-json-tmp\\Main.o: No such file or directory
`gcc.exe' failed in phase `Assembler'. (Exit code: 1)
Warning: 'c-sources: ../cbits/unescape_string.c' is a relative path outside of
the source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ..' is a relative path outside of the source tree.
This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../ffi' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../pure' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../attoparsec-iso8601' is a relative path outside of
the source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'hs-source-dirs: ../examples' is a relative path outside of the
source tree. This will not work when generating a tarball with 'sdist'.
Warning: 'include-dirs: ../include' is a relative path outside of the source
tree. This will not work when generating a tarball with 'sdist'.
Preprocessing executable 'aeson-benchmark-foldable' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-foldable' for aeson-benchmarks-0..
[1 of 1] Compiling Main             ( AesonFoldable.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-foldable\build\aeson-benchmark-foldable\aeson-benchmark-foldable-tmp\Main.o )
Preprocessing executable 'aeson-benchmark-map' for aeson-benchmarks-0..
Building executable 'aeson-benchmark-map' for aeson-benchmarks-0..
[1 of 1] Compiling Main             ( AesonMap.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-map\build\aeson-benchmark-map\aeson-benchmark-map-tmp\Main.o )

[Mistuke]

Right, that's not cabal failing but gccc, we have a ticket for that. The patch explicitly mentioned ghc is not excluded from MAX_PATH because of the use of gnu utilities. That's going to take some time to patch

…

On Fri, Jul 27, 2018, 10:00 AndreasPK ***@***.***> wrote: Building executable 'aeson-benchmark-compare' for aeson-benchmarks-0.. [1 of 1] Compiling Main ( AesonEncode.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-aeson-encode\build\aeson-benchmark-aeson-encode\aeson-benchmark-aeson-encode-tmp\Main.o ) [1 of 1] Compiling Main ( CompareWithJSON.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-compare-with-json\build\aeson-benchmark-compare-with-json\aeson-benchmark-compare-with-json-tmp\Main.o ) [ 1 of 10] Compiling Options ( Options.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-auto-compare\build\aeson-benchmark-auto-compare\aeson-benchmark-auto-compare-tmp\Options.o ) Assembler messages: Fatal error: can't create C:\\ghc\\msys64\\home\\Andi\\bench_script\\lib_bench\\aeson\\aeson_allCalls\\dist-vanilla\\build\\x86_64-windows\\ghc-8.7.20180718\\aeson-benchmarks-0\\x\\aeson-benchmark-auto-compare\\build\\aeson-benchmark-auto-compare\\aeson-benchmark-auto-compare-tmp\\Options.o: No such file or directory [2 of 7] Compiling Compare.JsonBuilder ( Compare\JsonBuilder.hs, C:\ghc\msys64\home\Andi\bench_script\lib_bench\aeson\aeson_allCalls\dist-vanilla\build\x86_64-windows\ghc-8.7.20180718\aeson-benchmarks-0\x\aeson-benchmark-compare\build\aeson-benchmark-compare\aeson-benchmark-compare-tmp\Compare\JsonBuilder.o ) `gcc.exe' failed in phase `Assembler'. (Exit code: 1) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABH3KanJrbX41JorOheFiwIrB5IRT9U0ks5uKtargaJpZM4Uj5ZE> .

[Mistuke]

Ultimately there's nothing cabal itself can do to fully fix this, for every workaround we do there will always be a corner case. I need to find some time to finish #4978 which should mitigate it for a bit but we need to fix the surrounding tools.

[Mistuke]

But thanks for the reminder @AndreasPK! I'll try to look at the remaining Linux failure this weekend.

[jasagredo]

I think this has been fixed already, there are no directories in store/ghc-xxx/incoming:

➜ ls -lah AppData/Local/cabal/store/ghc-9.8.2/incoming/ | grep "^d"
drwxr-xr-x 1 Javier Javier 0 Jul 18 23:30 .
drwxr-xr-x 1 Javier Javier 0 Jul 18 23:30 ..

I only see it with GHC 8.10.7.

I will preemptively close this issue. Feel free to re-open if you disagree.