compile requires zlib.h in default location

[MagicalTux]

When compiling rust-1.41.1 (x.py build) build fails if zlib.h is not present in a default location. Typically, pkg-config --cflags zlib would provide the appropriate location for zlib in case it is not in /usr/include, however rust does not make use of this. Also, editing config.toml to add include path in cflags or cxxflags under [llvm] will not solve this issue either.

I could find no documented way to alter cflags that works. During research I stumbled upon rust-lang/cargo#2112 which seems somewhat related, but altering RUSTFLAGS didn't solve this issue.

I expected to see this happen: Rust compiling successfully.

Instead, this happened: x.py build fails

Meta

python3 ./x.py build --exclude src/tools/miri:

[...]
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/build/dev-lang.rust/1.41.1/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libssh2-sys-4164d93400696d60/out/include" "-I" "libssh2/src" "-I" "/build/dev-lang.rust/1.41.1/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libssh2-sys-4164d93400696d60/out/build" "-I" "/pkg/main/dev-libs.openssl.core.1.1.1d/include" "-fvisibility=hidden" "-DHAVE_LONGLONG" "-DHAVE_SNPRINTF" "-DHAVE_UNISTD_H" "-DHAVE_INTTYPES_H" "-DHAVE_STDLIB_H" "-DHAVE_SYS_SELECT_H" "-DHAVE_SYS_SOCKET_H" "-DHAVE_SYS_IOCTL_H" "-DHAVE_SYS_TIME_H" "-DHAVE_SYS_UN_H" "-DHAVE_O_NONBLOCK" "-DLIBSSH2_OPENSSL" "-DHAVE_LIBCRYPT32" "-DHAVE_EVP_AES_128_CTR" "-DLIBSSH2_HAVE_ZLIB" "-o" "/build/dev-lang.rust/1.41.1/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libssh2-sys-4164d93400696d60/out/build/libssh2/src/crypt.o" "-c" "libssh2/src/crypt.c"
cargo:warning=libssh2/src/comp.c:41:11: fatal error: zlib.h: No such file or directory
cargo:warning=   41 | # include <zlib.h>
cargo:warning=      |           ^~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

In this specific context, zlib is not installed in the default path, but pkg-config --cflags zlib will return the appropriate flags to pass to cc for a successful build.

[nagisa]

The two things that depend on libssh2 are clippy and cargo.

[Tyg13]

From a cursory glance at its build script, looks like the libssh2-sys crate will attempt to build its own version of libssh2 from source if the environment variable LIBSSH2_SYS_USE_PKG_CONFIG is unset and (if it is set) if it is unable to find libssh2 via pkg-config. Presumably that's where the compile failed.

Assuming you have libssh2 available via pkg-config you can set LIBSSH2_SYS_USE_PKG_CONFIG, but that's undesirable for reasons mentioned in ssh2-rs#88.

Reading a bit further in build.rs, it looks like you can set DEP_Z_INCLUDE to the path where zlib.h is contained, and it will pass it as a -I to the cc invocation. That's probably your best bet here.

[MagicalTux]

Thank you very much for the response.

I've tried the LIBSSH2_SYS_USE_PKG_CONFIG method first, since we package a recent version of libssh2, however it seems that the cflags returned by pkg-config are not added to the compile command line, resulting in libssh2.h being not found.

exit code: 0
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/build/dev-lang.rust/1.41.1.linux.amd64/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libgit2-sys-f03983c628e87df7/out/include" "-I" "libgit2/src" "-I" "libgit2/deps/http-parser" "-I" "libgit2/deps/pcre" "-I" "/pkg/main/sys-libs.zlib.core.1.2.11/include" "-I" "/pkg/main/dev-libs.openssl.core.1.1.1d/include" "-fvisibility=hidden" "-DGIT_REGEX_BUILTIN=1" "-DHAVE_STDINT_H=1" "-DHAVE_MEMMOVE=1" "-DNO_RECURSE=1" "-DNEWLINE=10" "-DPOSIX_MALLOC_THRESHOLD=10" "-DLINK_SIZE=2" "-DPARENS_NEST_LIMIT=250" "-DMATCH_LIMIT=10000000" "-DMATCH_LIMIT_RECURSION=MATCH_LIMIT" "-DMAX_NAME_SIZE=32" "-DMAX_NAME_COUNT=10000" "-DSHA1DC_NO_STANDARD_INCLUDES=1" "-DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\"" "-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\"" "-o" "/build/dev-lang.rust/1.41.1.linux.amd64/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libgit2-sys-f03983c628e87df7/out/build/libgit2/src/transports/auth_negotiate.o" "-c" "libgit2/src/transports/auth_negotiate.c"
exit code: 0
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/build/dev-lang.rust/1.41.1.linux.amd64/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libgit2-sys-f03983c628e87df7/out/include" "-I" "libgit2/src" "-I" "libgit2/deps/http-parser" "-I" "libgit2/deps/pcre" "-I" "/pkg/main/sys-libs.zlib.core.1.2.11/include" "-I" "/pkg/main/dev-libs.openssl.core.1.1.1d/include" "-fvisibility=hidden" "-DGIT_REGEX_BUILTIN=1" "-DHAVE_STDINT_H=1" "-DHAVE_MEMMOVE=1" "-DNO_RECURSE=1" "-DNEWLINE=10" "-DPOSIX_MALLOC_THRESHOLD=10" "-DLINK_SIZE=2" "-DPARENS_NEST_LIMIT=250" "-DMATCH_LIMIT=10000000" "-DMATCH_LIMIT_RECURSION=MATCH_LIMIT" "-DMAX_NAME_SIZE=32" "-DMAX_NAME_COUNT=10000" "-DSHA1DC_NO_STANDARD_INCLUDES=1" "-DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\"" "-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\"" "-o" "/build/dev-lang.rust/1.41.1.linux.amd64/work/rustc-1.41.1-src/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/libgit2-sys-f03983c628e87df7/out/build/libgit2/src/transports/winhttp.o" "-c" "libgit2/src/transports/winhttp.c"
cargo:warning=libgit2/src/transports/ssh.c:11:10: fatal error: libssh2.h: No such file or directory
cargo:warning=   11 | #include <libssh2.h>
cargo:warning=      |          ^~~~~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

Note that pkg-config is providing the right cflags for libssh2, which are not used:

# pkg-config --cflags libssh2
-I/pkg/main/net-libs.libssh2.dev.1.9.0/include -I/pkg/main/dev-libs.openssl.core.1.1.1d/include -I/pkg/main/sys-libs.zlib.core.1.2.11/include
# ls /pkg/main/net-libs.libssh2.dev.1.9.0/include
libssh2.h  libssh2_publickey.h	libssh2_sftp.h

I will now try the DEP_Z_INCLUDE method too. By the way is there an equivalent variable to define the path for libz.so too (-L argument)?

[Tyg13]

If you check the source of the compilation, you'll see that you're getting a compilation error when building libgit2-sys, since the environment variable LIBGIT2_SYS_USE_PKG_CONFIG is not set. This causes libgit2-sys to attempt to build libgit2 from source.

This may sound incredibly familiar. You have two options -- either set LIBGIT2_SYS_USE_PKG_CONFIG if you have it available via pkg-config or set the necessary DEP_<LIB>_INCLUDE variables. In this case, DEP_OPENSSL_INCLUDE and DEP_SSH2_INCLUDE are probably relevant.

It sucks to have to keep playing dependency whack-a-mole. It may be worth to filing bugs with the maintainers of libgit2-sys and libssh2-sys to see if they can do something better for your system.

[Tyg13]

To answer your second question, I don't see equivalent environment variables to specify library paths, but it's likely that only the headers are necessary to compile the bundled versions of libssh2 and libgit2 provided with the crate.

[MagicalTux]

Rust is ready to roll.

It looks like setting DEP_Z_INCLUDE, DEP_OPENSSL_INCLUDE and DEP_SSH2_INCLUDE allowed for a successful build of rust 1.41.1.

I'm now going to test with LIBSSH2_SYS_USE_PKG_CONFIG and LIBGIT2_SYS_USE_PKG_CONFIG, and will also run tests directly with ssh2-rs in order to attempt to fix alexcrichton/ssh2-rs#169.

Should an issue with https://github.com/rust-lang/git2-rs also be created?

[Tyg13]

It would definitely be worthwhile, at least to make the issue known to them. Their build script is similar to ssh2-rs so I imagine it wouldn't be difficult to provide a similar patch.

[MagicalTux]

I believe it's OK to close this issue now, or after rust-lang/git2-rs#525 is merged.

Thank you very much @Tyg13 for your helpful comments and suggestions in solving this!

[Tyg13]

Of course! Glad to be of help.

[wez]

I don't think we've really fixed anything; the libssh2 PR caused other breakages; discussion is on alexcrichton/ssh2-rs#174

[wez]

I believe that a proper fix for this requires changes in the libz-sys crate to ensure that it exports DEP_Z_INCLUDE or otherwise defines the mechanism by which downstream crates can locate the needed headers.

If the official way to deal with this is to use pkg-config then care needs to be taken to disable the automatic emission of cargo options using:
https://docs.rs/pkg-config/0.3.17/pkg_config/struct.Config.html#method.cargo_metadata
to avoid accidental linkage to system libraries when something like -L/usr/lib/x86_64-linux-gnu is added as a side effect.

[jyn514]

It looks like this is an upstream bug in libz-sys, so I'm going to close this. Feel free to reopen it on their issue tracker.