This builder is setup using Gentoo toolchains and system image.
- /src is the directory for the kernel sources git repo, mounted read-only
- /build is the checked out kernel and temporary output (tmpfs recommended, but can grow quite large)
- /install is where the kernels are installed to, currently unused
- /logs are where logs are written out to, emails into /logs/emails
Since this setup makes it easy to produce toolchains and userland
for several of the x86_64 sub-ABIs, I've experimented a bit with x86
(32-bit) vs x86_64 vs x32. I've also made -march=native
builds to
see if running with generically tuned toolchains leaves performance on
the floor on my hardware.
My hardware is a 32-core Threadripper 2990WX, with 64GB RAM and NVMe (Samsung 980 EVO).
Without including all performance data, in general most runs are along the below runtimes. There is some noise in the data from run to run, but most of them come out around these relative ballparks when building both 32- and 64-bit targets:
- Generic x86_64: 2419 seconds time elapsed
-mtune-native
: 2386 seconds time elapsed- Generic x86: 2392 seconds time elapsed
- Generic x32: 2171 seconds time elapsed
So:
- Building with
-march=native
buys ~1% - Building with a x86 toolchain buys ~5-6%
- Building with a x32 toolchain buys ~10%
Conclusion: The choice for me right now is quite simple. I am using the x32 docker image for the main builder workload.
docker build -t local/stage3-native -f Dockerfile.gentoo-base .
docker build -t local/builder .
I've gotten lazy and written an ugly script for it. A Makefile would be nicer but since Docker caches intermediate layers it's not quite as needed.
./build-images.sh
A few of the polling-loop hacks I have aren't in this repo, but this is how they launch the container:
#!/bin/bash
[...]
container="local/kernelbuilder"
# build <remote> <ref>
build() {
CNTR="${container}"
case $2 in
v3*)
CNTR="${container}-7.3.0"
;;
v4.[2-9]*)
CNTR="${container}-7.3.0"
;;
v4.1[0-8]*)
CNTR="${container}-7.3.0"
;;
esac
docker run -t --mount type=bind,src=/home/build/work/batch,dst=/src,ro \
--mount type=bind,src=/ds/logs/buildlogs,dst=/logs \
--tmpfs /build:size=60G,exec \
--net none \
-e ARCH="riscv x86 arm arm64" \
${CNTR} \
"$1/$2"
}