Cross compiling with sysroot

[andber1]

I have an armv7-unknown-linux-gnueabihf target with a rather old glibc 2.20. So the standard way of cross compiling does not work for me, because the glibc versions provided by the ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf images do not match my target system. So my naive idea was, why not link against the target system itself? I have an image of the target system that I have mounted into the docker image so that I can tell the linker to use it as sysroot. I am not an expert when it comes to cross compiling, so maybe this idea does not work at all. It would be nice, if someone can help me.

These are my attempts so far:

Sysroot link attempts

I added these lines to Cross.toml in order to mount my sysroot and configuring the linker to use it:

[build]
default-target = "armv7-unknown-linux-gnueabihf"
[build.env]
volumes = ["MY_SYSROOT=/path/to/my/sysroot"]
passthrough = ["CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_RUSTFLAGS=-C link-arg=--sysroot=/path/to/my/sysroot"]

Unfortunately I cannot use the environment variable MY_SYSROOT in the last line, because it would no be expanded. Off-topic question: Is there a simple solution for that?
I noticed that the linker does not use the libc from /path/to/my/sysroot/lib/libc-2.20.so (it linked without errors, but the resulting executable has GLIBC links greater 2.20 when examining with readelf). So I explicitly tell the linker to use that version by adding this line in build.rs:

println!("cargo:rustc-link-lib=c-2.20");

I asume that the linker uses libc from my sysroot now, but it still uses a different libpthread that is not compatible. For example I get the linker error:

/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/lib/../lib/libpthread.so: undefined reference to `_IO_enable_locks@GLIBC_PRIVATE'

What is the correct way to tell the linker to only look in my sysroot? Apparently it still looks in /usr/lib/gcc-cross/arm-linux-gnueabihf.
I also tried to specify the search paths in build.rs in addition.

println!("cargo:rustc-link-search=/path/to/my/sysroot/lib/");
println!("cargo:rustc-link-search=/path/to/my/sysroot/usr/lib/");

Then I get:

arm-linux-gnueabihf/bin/ld: cannot find /usr/lib/libpthread_nonshared.a inside /path/to/my/sysroot

libpthread_nonshared.a is not available on my target system. I am not sure why this happens. Can anyone explain this?

Creating new image with correct libc version

I tried to create a new docker image that has the correct libc version. I followed the instructions in the cross-toolchains repository and in this discussion. I did manage to create a new image using these commands:

xtask configure-crosstool armv7-unknown-linux-gnueabihf-cross --glibc-version 2.20 --gcc-version 4.9.1 --linux-version 3.0.35
xtask build-docker-image armv7-unknown-linux-gnueabihf-cross

In the end there was one warning. I am not sure if that can be ignored:

[Warning] one or more build args were not consumed: [CROSS_TARGET_TRIPLE]
Successfully tagged ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf-cross:local

I was able to successfully compile a hello world program (without setting a sysroot for now). But it was using a new libc. And looking into the image I found this file: /usr/lib32/libc-2.31.so. Shouldn't that be 2.20?

Using zig

Now I tried to specifiy the correct libc version with zig. I added the following to my Cross.toml:

[build]           
zig = "2.20"

The sysroot was still set. The program compiled and linked successfully. But I get a core dump when running the executable on the target:

Core was generated by `./hello-world'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x4015063c in ?? () from /lib/ld-linux-armhf.so.3
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file hello-world.

Does anyone have an idea what went wrong?

Using musl target

Using the target armv7-unknown-linux-musleabihf works out of the box. However, I am curious why my attempts above do not work. Maybe someone has an idea?

[Emilgardis]

I was able to successfully compile a hello world program (without setting a sysroot for now). But it was using a new libc. And looking into the image I found this file: /usr/lib32/libc-2.31.so. Shouldn't that be 2.20?

No, the sysroot is elsewhere (/x-tools/), not in /usr/

I'm not sure what went wrong with zig, as for the attempt to borrow the sysroot from the target, not going to delve into it :3

[andber1]

Thanks for the clarification. But even the libc version in x-tools is not 2.20 as I have specified (and the hello-world executable also uses the wrong libc).

$ /x-tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot/usr/bin/ldd --version
ldd (crosstool-NG UNKNOWN) 2.38

Do you have any idea what went wrong when creating the new image? I have used this file:

crosstool-ng/armv7-unknown-linux-gnueabihf-cross.config.in

CT_CONFIG_VERSION="4"
CT_PREFIX_DIR="/x-tools/${CT_TARGET}"
CT_DOWNLOAD_AGENT_CURL=y
CT_ARCH_ARM=y
CT_ARCH_ARM_MODE_THUMB=y
CT_ARCH_SUFFIX="v7"
CT_ARCH_ARCH="armv7-a"
CT_ARCH_FPU="vfpv3-d16"
CT_ARCH_FLOAT_HW=y
CT_KERNEL_LINUX=y
%CT_LINUX_V%
%CT_LINUX%
CT_BINUTILS_V_2_32=y
%CT_GLIBC_V%
%CT_GLIBC%
%CT_GCC_V%
%CT_GCC%
CT_CC_LANG_CXX=y
CT_GETTEXT_V_0_19_8_1=y
CT_GMP_V_6_1=y
CT_ISL_V_0_20=y
CT_LIBICONV_V_1_15=y
CT_NCURSES_V_6_1=y

After this command

xtask configure-crosstool armv7-unknown-linux-gnueabihf-cross --glibc-version 2.20 --gcc-version 4.9.1 --linux-version 3.0.35

this file was created:

crosstool-config/armv7-unknown-linux-gnueabihf-cross.config

CT_CONFIG_VERSION="4"
CT_PREFIX_DIR="/x-tools/${CT_TARGET}"
CT_DOWNLOAD_AGENT_CURL=y
CT_ARCH_ARM=y
CT_ARCH_ARM_MODE_THUMB=y
CT_ARCH_SUFFIX="v7"
CT_ARCH_ARCH="armv7-a"
CT_ARCH_FPU="vfpv3-d16"
CT_ARCH_FLOAT_HW=y
CT_KERNEL_LINUX=y
CT_LINUX_V_3_0=y
# CT_LINUX_NO_VERSIONS is not set
CT_LINUX_VERSION="3.0.35"
CT_LINUX_older_than_4_8=y
CT_LINUX_4_8_or_older=y
CT_LINUX_older_than_3_7=y
CT_LINUX_3_7_or_older=y
CT_LINUX_older_than_3_2=y
CT_LINUX_3_2_or_older=y
CT_BINUTILS_V_2_32=y
CT_GLIBC_V_2_20=y
# CT_GLIBC_NO_VERSIONS is not set
CT_GLIBC_VERSION="2.20"
CT_GLIBC_2_29_or_older=y
CT_GLIBC_older_than_2_29=y
CT_GLIBC_2_27_or_older=y
CT_GLIBC_older_than_2_27=y
CT_GLIBC_2_26_or_older=y
CT_GLIBC_older_than_2_26=y
CT_GLIBC_2_25_or_older=y
CT_GLIBC_older_than_2_25=y
CT_GLIBC_2_24_or_older=y
CT_GLIBC_older_than_2_24=y
CT_GLIBC_2_23_or_older=y
CT_GLIBC_older_than_2_23=y
CT_GLIBC_2_20_or_older=y
CT_GLIBC_2_17_or_later=y
CT_GLIBC_later_than_2_14=y
CT_GLIBC_2_14_or_later=y
CT_GCC_V_4=y
# CT_GCC_NO_VERSIONS is not set
CT_GCC_VERSION="4.9.1"
CT_CC_LANG_CXX=y

CT_GETTEXT_V_0_19_8_1=y

CT_GMP_V_6_1=y

CT_ISL_V_0_20=y

CT_LIBICONV_V_1_15=y

CT_NCURSES_V_6_1=y

After

xtask build-docker-image armv7-unknown-linux-gnueabihf-cross

I got a new image. So basically I followed the instructions in this discussion as already mentioned.
But the libc version is wrong. Do you see the mistake?

Regarding the sysroot issue: I can understand if you do not want to delve into it. However, can you tell me if you see a general flaw in my idea? It should be possible to link against libc from my sysroot, right?

[Emilgardis]

ldd --version only shows what was linked when it was built, you'll need to check the .so
strings /x-tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot/usr/lib/libc.so | grep "GLIBC"

Yes, theorethically what you did with copying could work, what you would have to do is undo the config we've done with env-vars