ESPs on multi disk pool

[Sithuk]

What is the correct approach to configure the multiple EFI system partitions (ESP) for a zfsbootmenu installation on a multi disk mirror pool? For example, at the moment I mount a single ESP partition from one of the disks to /boot/efi and generate-zbm creates and saves the necessary EFI boot files in the relevant ImageDir. However, this is not a resilient approach in the event that the disk fails. I would like the loss of a single drive to not affect the ability of the mirror pool system to boot.

How should the relevant files also be copied / mirrored / sync'd to the other ESP partitions? Will I have to create a post generate-zbm script to mount and sync the files to the other ESPs? Would such a script be best placed in the PostHooksDir? Or is there a better approach?

Related ubuntu zfsbootmenu install script issue here:
Sithuk/ubuntu-server-zfsbootmenu#12

[ahesford]

I use an mdraid mirror with version 1.0 metadata. The version is important. Newer mdraid versions add metadata at the front of the block devices, so they won't be recognized as normal filesystems by your firmware. Versions 1.0 and below store metadata at the end; each block device on a mirror will look like an independent, valid filesystem to the firmware.

Some people worry about this configuration because your firmware may write to the filesystem it uses and cause inconsistencies. I think this concern is overblown because we're talking about an EFI system partition. First, resilvering a 500-MB partition should they become inconsistent is a trivial operation. Second, the EFI partition is effectively disposable anyway. If there are problems that prevent boot, the worst possible outcome is that you'll need to dump a ZBM EFI executable to a USB drive for recovery.

If you don't like the mdraid approach, we have a contrib script to sync multiple ESPs. Adapt this to your needs and you're all set.

[Sithuk]

Perfect, thank you. What do you see as the advantage of the mdraid approach over the contrib script given the risk you mentioned about firmware overwrites? Would it be that the fstab entry for /boot/efi will work seamlessly in the event of a drive failure because it will reference the MD array as a whole (e.g. md0) rather than a specific ESP?

[edit: I can see Lennart Poettering has recently created a blog post on this topic (see "Addendum: You got RAID"):
https://0pointer.net/blog/linux-boot-partitions.html [3 November 2022]

His recommendation appears to be to use the userland tooling "contrib scipt" approach.
"My recommendation would be to solve this via userspace tooling. If redundant disk support shall be implemented for the ESP, then create separate ESPs on all disks, and synchronize them on the file system level instead of the block level. Or in other words, the tools that install/update/manage kernels or boot loaders should be taught to maintain multiple ESPs instead of one. Copy the kernels/boot loader files to all of them, and remove them from all of them. Under the assumption that the goal of RAID is a more reliable system this should be the best way to achieve that, as it doesn’t pretend the firmware could do things it actually cannot do. Moreover it minimizes the complexity of the boot loader, shifting the syncing logic to userspace, where it’s typically easier to get right."

There is also a systemd github issue here:
systemd/systemd#19668 ]

[edit2: I can see the ZBM team has already commented on Poettering's post on the following HackerNews thread:
https://news.ycombinator.com/item?id=33455052 ]

[ahesford]

The Poettering article makes no sense. He seems to be motivated by the fact that mounting /boot/efi under /boot, itself a separate mount point, causes problems for systemd mount logic. The rest of the article seems to be a bunch of non sequiturs as made-up "problems" that systemd somehow resolves. In reality, no problems are solved because they never existed in the first place.

I adopted the mdraid approach before zdykstra wrote the contrib script. Because there is minimal risk, I've had no motivation to redo my approach.

[zdykstra]

On my system, I don't even bother with redundant ESPs. I attached an 8GB USB drive to a port on my motherboard. Since it's rarely ever written to or read from, it should last for ages.

[Nukien]

Yeah, but if it fails for whatever reason, then you're rebuilding things. mdadm mirror for all ESP partitions, with /dev/md126 or whatever mounted at /boot/efi and it all just works.

I have a setup script for multi-disk root-on-zfs working cleanly, supporting both uefi booting as well as legacy with syslinux. Disk(s) can be moved between uefi or legacy bios systems and they boot fine. syslinux and rEFInd both live under /boot/efi/EFI (we chatted about this in #zfsbootmenu a while back)