Migrating from 4.16.0-0.okd-2024-03-27-015536 to 4.16.0-0.okd-scos-2024-08-01-132038

[wouterhummelink]

I thought it would be good to share my experience.

I run an OKD SNO cluster in my homelab on a DL360G8.
Since I was running a 4.16 FCOS release that was recently pruned I was left without an upgrade path.
So I decided to attempt a force upgrade to the latest version in 4-scos-next stream. The update ran smoothly, up until the reboot.

After the reboot rpm-ostreed would not start due to a dependency failing with the fsck job for the root volume failing. That turned out to be caused by an ext4 feature used by newer kernels.
I fixed this by compiling e2fsprogs from f39 in a stream9 vm and installing those packages in a temporary installroot, mounted dev in there and chrooted to run tune2fs to remove the unsupported feature.

tune2fs -O ^orphan_file /dev/disk/by-uuid/7cac3093-100d-48cd-85e3-125db4adf9db

I also had another issue that is less easily explained...
I have the LVM CSI Operator running, the volume group managed by it was not coming up properly due to 2 missing PVs, however, it only ever had one PV. I removed the offending PVs and all is well now.

[ausil]

I wonder if you could have turned that feature off from fcos before upgrading?

[ArthurVardevanyan]

Ran into the same issue, for me it didn't effect any node that started life on FCOS 38, and my control plane was fine, so I repaved any node that was effected back to FCOS 38

[GingerGeek]

I believe this is the same issue as #2041 related to newer ext4 enabled by default in FCOS >39 which can prevent Pivots in certain circumstances. I'll close this discussion in favour of the open issue, however please let us know if this is incorrect.