Failed upgrade from 4.15.0-0.okd-2024-02-10-035534 to 4.16.0-0.okd-scos-2024-09-24-151747 (e2fsprogs-1.47.0)

[forzamehlano]

Upgrading an IPI cluster on AWS using STS from 4.15.0-0.okd-2024-02-10-035534 to 4.16.0-0.okd-scos-2024-09-24-151747.

Followed the steps as per https://github.com/okd-project/okd-web/pull/22/files along with the standard STS upgrade steps and all was well up until the mco rollout.

The master nodes updated fine, but our worker nodes did not. They appear to be hitting the same issue as #1997 and the machine-config-daemon sits in a crashloopbackoff with:

I1029 12:57:16.538760 8010 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2842-g6a425abb-dirty (6a425abbe5e330f0e7bc0849f4927d4c55d0c9e5)
I1029 12:57:16.538863 8010 update.go:2618] Running: mount --rbind /run/secrets /rootfs/run/secrets
I1029 12:57:16.541023 8010 update.go:2618] Running: mount --rbind /usr/bin /rootfs/run/machine-config-daemon-bin
I1029 12:57:16.542711 8010 daemon.go:517] using appropriate binary for source=rhel-9 target=rhel-9
I1029 12:57:16.581316 8010 daemon.go:570] Invoking re-exec /run/bin/machine-config-daemon
I1029 12:57:16.632818 8010 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2842-g6a425abb-dirty (6a425abbe5e330f0e7bc0849f4927d4c55d0c9e5)
E1029 12:57:16.632987 8010 rpm-ostree.go:276] Merged secret file does not exist; defaulting to cluster pull secret
I1029 12:57:16.633024 8010 rpm-ostree.go:263] Linking ostree authfile to /var/lib/kubelet/config.json
F1029 12:57:16.733526 8010 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

Digging into the OS, the following error is seen:

Oct 29 12:55:45 systemd-fsck[7668]: /dev/nvme0n1p3 has unsupported feature(s): FEATURE_C12
Oct 29 12:55:45 systemd-fsck[7668]: e2fsck: Get a newer version of e2fsck!
Oct 29 12:55:45 systemd-fsck[7668]: boot: ********** WARNING: Filesystem still has errors **********
Oct 29 12:55:45 systemd-fsck[7665]: fsck failed with exit status 12.
Oct 29 12:55:45 systemd[1]: systemd-fsck@dev-disk-by\x2duuid-80b36fde\x2d67a9\x2d474b\x2da0cf\x2d8d2d95521b9a.service: Main process exited, code=exited, status=1/FAILURE

We're using machinesets so I just scaled down this particular machineset and back up again so it'd create a new machine but the same error exists on the new node.

So it would seem I'm unable to create new machines at the SCOS level due to this filesystem issue.

The issue I presume is that FCOS39 includes e2fsprogs-1.47.0 where as SCOS only includes 1.46.x which doesn't include these new features.

https://gitlab.alpinelinux.org/alpine/aports/-/issues/15210 appears to have some background info on the change that's causing this.