Swap fails to mount due to a race condition

[alehed]

Bug

Usually on the first run of ignition it fails with the swap device unit timing out. After that, if I power-cycle, on the second boot Ignition will go though without errors.

The problem seems to be due to a race condition where the boot disk preparation doesn't complete before the swap unit is expected to be there. Note that sometimes the bug doesn't happen, but usually it does.

Operating System Version

I am running ignition on the latest CoreOS version. The Ignition file was compiled from a Butane file on Linux with Butane version v0.18.0.

Ignition Version

2.16.2

Environment

I am running Ignition on bare metal on a Raspberry Pi 3 (it also happens on the Raspberry Pi 4) in Fedora CoreOS (38.20230918.3.2).

Expected Behavior

Mounting swap succeeds on first boot.

Actual Behavior

The swap device fails to mount causing the first boot to fail and present a recovery shell.

Reproduction Steps

1. Run coreos-installer on an instance with two disks giving it the ignition file from below.
2. Reboot

Other Information

The following ignition file can be used to reproduce this:

{
  "ignition": {
    "version": "3.4.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-ed25519 ...."
        ]
      }
    ]
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/disk/by-id/coreos-boot-disk",
        "partitions": [
          {
            "label": "root",
            "number": 4,
            "resize": true,
            "sizeMiB": 8192
          },
          {
            "label": "swap",
            "number": 5,
            "sizeMiB": 4096
          }
        ],
        "wipeTable": false
      },
      {
        "device": "/dev/disk/by-id/usb-...",
        "partitions": [
          {
            "label": "var",
            "number": 1
          }
        ],
        "wipeTable": false
      }
    ],
    "filesystems": [
      {
        "device": "/dev/disk/by-partlabel/swap",
        "format": "swap"
      },
      {
        "device": "/dev/disk/by-partlabel/var",
        "format": "ext4",
        "path": "/var"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "contents": "# Generated by Butane\n[Swap]\nWhat=/dev/disk/by-partlabel/swap\n\n[Install]\nRequiredBy=swap.target",
        "enabled": true,
        "name": "dev-disk-by\\x2dpartlabel-swap.swap"
      },
      {
        "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var.service\n\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nType=ext4\n\n[Install]\nRequiredBy=local-fs.target",
        "enabled": true,
        "name": "var.mount"
      }
    ]
  }
}

I didn't manage to reproduce this with a single disk, but due to it being a race condition and not always happening, I cannot say for certain it doesn't also happen without the separate var device.

The rdsosreport.txt looks like this.

[jlebon]

The interesting bit is here:

[   19.519354] localhost ignition[820]: disks: createPartitions: op(4): op(6): [started]  deleting 1 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/coreos-boot-disk"
[   19.534201] localhost ignition[820]: disks: createPartitions: op(4): op(6): executing: "sgdisk" "--delete=4" "--new=4:1050624:+16777216" "--change-name=4:root" "--typecode=4:0FC63DAF-8483-4772-8E79-3D69D8477DE4" "--partition-guid=4:93851DF3-8B9E-4196-90C6-C95B321BA51B" "--new=5:0:+8388608" "--change-name=5:swap" "/run/ignition/dev_aliases/dev/disk/by-id/coreos-boot-disk"
[   19.589134] localhost systemd-journald[256]: Missed 3 kernel messages
[   19.586092] localhost kernel:  sda: sda1 sda2 sda3 sda4
[   20.750518] localhost ignition[820]: disks: createPartitions: op(4): op(6): [finished] deleting 1 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/coreos-boot-disk"

Notice how the kernel partition reread didn't pick up sda5. Hmm, this might actually be fixed by #1717.

[alehed]

Ah, nice, I will retest once this change lands in CoreOS.

[jlebon]

We merged #1717 now, but re-reading this I'm not actually sure if it'll fix the issue here. It's not clear why exactly the device is timing out. It might be BLKRRPART failing (and sgdisk normally prints a warning when that happens, but Ignition swallows its output since overall it passed), but it's not clear why that would even fail since nothing should be using the disk at this point of the boot.

[jlebon]

Are you still able to reproduce this? I can't. (This is with new-ish FCOS, without #1717).