FreeBSD all architectures via jail and example

[am11]

Background

At present, there is only FreeBSD amd64 image available by GCE and CirrusCI. To be able to configure CI/CD for other architectures, user has option to resort to cross-compilation scenarios.

Problem Statement

Unlike Linux, where we can set up a cross-compiler toolchain and build binaries for other architectures, I could not find a way to make GCC or Clang to produce 32-bit binaries on FreeBSD with lib32.txz and -m32 compiler switch. Although not clearly documented, but it appears like FreeBSD does not support cross-compilation and instead recommends jail(2) as a solution to such scenarios.

The creation of jail is not straightforward either. There is a native way, e.g. https://gundersen.net/32bit-jail-on-64bit-freebsd/, and then there are some wrapper tools from ports that provide the ability to create jails: iocell, iocage, ezjail, cbsd etc. In my little research, none of these tools are natural fit (do not scale well) for most common CI/CD build pipelines where the basic scenarios are: build, test and upload artifacts (to GitHub etc.). Some methods/tools require manual configurations for network, ZFS pool etc., others are yet to support cross architectures.

Proposal 1

In theory, to build xarch 32-bit binaries on CirrusCI, user can configure FreeBSD jail for ia32. Similarly, arm and Aarch64 archs could be entertained.

Since it takes a lot of effort to get a basic jail up and working even on a baremetal (unless user is area expert in FreeBSD jails and skels); it would be very useful if docs could reflect a .cirrus.yml example on how to create and cache i386/ARM jail environments, so community can effectively make use of it in common CI/CD scenarios.

Proposal 2

It would be even more efficient and less error-prone for consumers, if Cirrus Labs could instead provide a slim abstraction over 'cached' jails, to avoid downloading of base.txz and caching it in varied (suboptimal) ways. This way, the downstreamers would be able to do something like:

freebsd_instance:
  image: freebsd-11-2-release-amd64

freebsd_build_amd64_task:
...

freebsd_instance:
  image: freebsd-11-2-release-amd64
  target_architecture: i386

freebsd_build_i386_task:
...

freebsd_instance:
  image: freebsd-11-2-release-amd64
  target_architecture: arm

freebsd_build_arm_task:
...

freebsd_instance:
  image: freebsd-11-2-release-amd64
  target_architecture: aarch64

freebsd_build_aarch64_task:
...

upload_artifacts_task:
...

[fkorotkov]

If I understand you correctly proposal 2 is basically a syntactic sugar for proposal 1. Basically do the same just with target_architecture config.

Have you tried to create jails? I have very little experience with FreeBSD and if you can help me get a working prototype of proposal 1 then we can add as a target_architecture config or something like this.

[asomers]

Actually, these proposals are both just a half-baked version of a superior feature: run CI in jails instead of VMs. Jails are good for CI applications because they very low startup time. Also, both 64-bit and 32-bit jails can run on the same host. Older FreeBSD versions can also run in jails. However, GCE doesn't have any jail support. If Cirrus wants to go this route, then you'll have to setup your own jail server (it can run in a GCE VM) and spin jails up and down as needed. Whether you choose to put in the effort basically boils down to how much you value the reduced startup time. It's a judgement call.

[fkorotkov]

@asomers got it! Thank for the detailed explanation.

Seems right now it doesn't worth the effort since it will affect few people but will require development and most importantly maintenance. Will keep this issue in mind but will focus the effort on other issues that affect more users.

[RDIL]

IMO, Cirrus has the fastest startup time for all the VMs except macOS. That is one of the greatest things about it. I don’t really see this as an issue.

[asomers]

BTW, to answer the original question, @am11 should probably just use a 32-bit VM image, assuming GCE supports 32-bit VMs.

[am11]

@fkorotkov, sorry for the late reply. I do not have a 100% working example with jail yet, was trying to finish one before getting back to you. :)

Many thanks to @olevole for continuous help: cbsd/cbsd#367. So far I have a WIP branch:
sass/node-sass@master...am11:freebsd-ci (heredoc syntax is not working Cirrus YAML, as it does in Travis?). It is bit messy, i will clean it up a bit once i get ia32 binaries compiling and upload to GH artifacts.

In my short experience with FreeBSD, the main pain point is that creation of a single jail non-interactively with working network requires a lot of effort compared to competing technologies like docker and lxc. Ballpark: it is 80x more complex than just doing docker run -it alpine or lxc-create -n mygen -t gentoo; lxc-start -n mygen to get started with a fully functional isolated environment in no time. Hence, I think FreeBSD folks did not make jails (or even jail wrappers) easy enough for regular or power users to fall into the pit of success, with sane defaults, and basic options like non-interactive usage, always-on-and-connected network etc. Perhaps it is not their design goal.

As mentioned earlier, GCE only provides amd64 image for FreeBSD. Neither ia32 image, nor arm/aarch64 are available. Thus, having the jail capability in Cirrus infra would open avenues for many downstream users.

[fkorotkov]

Seems someone successfully used jails here: sass/node-sass#2592

I'm not that qualified to judge how accurate it is. @am11 @asomers PTAL.

[fkorotkov]

Oh. @am11 it was you! 😅I got too deep into reviewing the scripts and didn't notice. My bad!

[am11]

LOL, I was going to write something like "as promised, here is the living example..." 😹
Yup, some parts could be simplified further, if we get semantically nicer a way to specify target architecture, and jail creation and handling takes place behind the wraps.

---

(most likely i was doing it wrong..)
Aside, for some reason publish task didn't work for me, so i added publish_script per task. What I wanted was to somehow retain the output of each dependee job (vendor directory in our case), so publish task could loop through the combined directory and upload artifacts.

[fkorotkov]

Awesome! I'll check it more precisely tomorrow to see how it can be generalized. 🙌