overlay.d: add service to check for unique-boot after ignition

[nikita-dubrovskii]

After ignition's stages partition table could be modified, and therefore
kernel may have an old picture of disks and partitions. During verification
we force kernel to reread the partition table for each disk and call udevsettle,
which in turn leads to modification of "/dev/disk/by-/" symlinks and systemd
may send SIGTERM to services with "Require=dev-disk-by\x2dlabel-boot.device".
To avoid that this checks for the unique fs labeled 'boot' after ignition's
disks stage was finished but before mounting stage.

Issue: coreos/fedora-coreos-tracker#1105

[cgwalters]

I didn't do an in-depth analysis here, but this seems sane to me.

[jlebon]

Thanks @nikita-dubrovskii and @sohankunkerkar!

[jlebon]

coreos.boot-mirror is failing with

Starting CoreOS Ensure Unique Boot Filesystem...
Error: System has 0 devices with a filesystem labeled 'boot': []
coreos-ignition-unique-boot.service: Main process exited, code=exited, status=1/FAILURE

which is interesting. This runs right after ignition-disks which does do udevadm settle. And --rereadpt also does a udevadm settle, though based on this it's possible we didn't actually wait for anything at all.

Hmm, but actually I think we could have verify-unique-fs-label just keep waiting until the label shows up after re-reading all partition tables. This would be the equivalent of Requires=dev-disk-by\x2dlabel-boot.device and After=dev-disk-by\x2dlabel-boot.device, except we can't do that directly otherwise we'll hit the exact issue this PR intended to solve.

[nikita-dubrovskii]

using this with coreos/coreos-installer#813 fixes CI (locally)

[jlebon]

This is pending a new coreos-installer release.

[jlebon]

We have new enough coreos-installer now, so this is unblocked.
LGTM!

[jlebon]

I restarted CI here to make sure it tests on top of the latest. Let's rerun it multiple times before merging this to try to make sure we're not introducing flakiness again. Edit: I've now enabled auto-merge after restarting the test three times.

Ideally, we should add tests for both coreos-unique-boot.service and this new service. I think we already have some tests in kola which expect failure in the initrd. OK, filed coreos/coreos-assembler#2953 for this.