Worker Kernel Panic: VFS: Busy inodes after unmount of ceph (ceph)

[alexiusflavius]

Under moderate / high load worker nodes were rebooting
Testing OKD on Openstack I acme across an issue where workers would just reboot for no apparent reason,
Checking worker logs the stayed on Openstack side I found that panic is caused by busy inodes,
Everything is working fine on the production cluster running 4.12.0-0.okd-2023-02-18-033438,
Pods are running on Rocky 8.8 and 9.2 with the same issue,

[97931.567375] VFS: Busy inodes after unmount of ceph (ceph)
[97931.568295] kernel BUG at fs/super.c:502!
[97931.571898] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[97931.572476] CPU: 21 PID: 1581298 Comm: umount Not tainted 6.4.7-200.fc38.x86_64 #1
[97931.573162] Hardware name: RDO OpenStack Compute/RHEL, BIOS 1.16.1-1.el9 04/01/2014
[97931.573833] RIP: 0010:generic_shutdown_super+0x142/0x150
[97931.574372] Code: cc cc e8 81 cf f1 ff 48 8b bb f8 00 00 00 eb db 48 8b 43 28 48 8d b3 c8 03 00 00 48 c7 c7 98 c6 8f 84 48 8b 10 e8 ce b3 d4 ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90
[97931.576013] RSP: 0018:ff77681d82c6be10 EFLAGS: 00010246
[97931.576528] RAX: 000000000000002d RBX: ff49cf134fbcc000 RCX: 0000000000000000
[97931.577174] RDX: 0000000000000000 RSI: ff49cf3007761540 RDI: ff49cf3007761540
[97931.577836] RBP: ffffffffc0bec3a0 R08: 0000000000000000 R09: ff77681d82c6bcb8
[97931.578468] R10: 0000000000000003 R11: ff49cf307fd3bb28 R12: ff49cf14b1de4ccc
[97931.579115] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[97931.579759] FS:  00007f2a4e061800(0000) GS:ff49cf3007740000(0000) knlGS:0000000000000000
[97931.580554] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[97931.581318] CR2: 00007f2a4e1bb620 CR3: 0000000218662006 CR4: 0000000000771ee0
[97931.581960] PKRU: 55555554
[97931.582310] Call Trace:
[97931.582651]  <TASK>
[97931.582968]  ? die+0x36/0x90
[97931.583326]  ? do_trap+0xda/0x100
[97931.583717]  ? generic_shutdown_super+0x142/0x150
[97931.584183]  ? do_error_trap+0x6a/0x90
[97931.584583]  ? generic_shutdown_super+0x142/0x150
[97931.585055]  ? exc_invalid_op+0x50/0x70
[97931.585454]  ? generic_shutdown_super+0x142/0x150
[97931.585914]  ? asm_exc_invalid_op+0x1a/0x20
[97931.586334]  ? generic_shutdown_super+0x142/0x150
[97931.586791]  kill_anon_super+0x18/0x30
[97931.587177]  ceph_kill_sb+0x3a/0x80 [ceph]
[97931.587593]  deactivate_locked_super+0x30/0xa0
[97931.588029]  cleanup_mnt+0xba/0x150
[97931.588388]  task_work_run+0x5a/0x90
[97931.588758]  exit_to_user_mode_prepare+0x1e6/0x1f0
[97931.589199]  syscall_exit_to_user_mode+0x1b/0x40
[97931.589622]  do_syscall_64+0x6c/0x90
[97931.589989]  ? syscall_exit_to_user_mode+0x1b/0x40
[97931.590416]  ? do_syscall_64+0x6c/0x90
[97931.590783]  ? exc_page_fault+0x7f/0x180
[97931.591156]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[97931.591590] RIP: 0033:0x7f2a4e27858b
[97931.591973] Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 49 08 0c 00 f7 d8
[97931.593389] RSP: 002b:00007ffec41f5718 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[97931.594036] RAX: 0000000000000000 RBX: 000055e9d5230bd0 RCX: 00007f2a4e27858b
[97931.594638] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055e9d5230e00
[97931.595239] RBP: 00007ffec41f57f0 R08: 0000000000000073 R09: 0000000000000001
[97931.595806] R10: 0000000000000000 R11: 0000000000000246 R12: 000055e9d5230ce8
[97931.596367] R13: 0000000000000000 R14: 000055e9d5230e00 R15: 0000000000000000
[97931.596932]  </TASK>
[97931.597183] Modules linked in: veth ipt_REJECT nf_reject_ipv4 xt_nat nf_conntrack_netlink xt_CT tls ceph libceph fscache netfs xt_REDIRECT xt_MASQUERADE nft_chain_nat xt_mark xt_conntrack xt_comment nft_compat nf_tables rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay isofs intel_rapl_msr iTCO_wdt intel_pmc_bxt iTCO_vendor_support i2c_i801 intel_rapl_common cirrus lpc_ich i2c_smbus virtio_balloon joydev xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic virtio_net ghash_clmulni_intel sha512_ssse3 net_failover virtio_blk failover serio_raw scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables ip_tables dm_multipath fuse qemu_fw_cfg
[97931.604244] ---[ end trace 0000000000000000 ]---
[97931.604716] RIP: 0010:generic_shutdown_super+0x142/0x150
[97931.605188] Code: cc cc e8 81 cf f1 ff 48 8b bb f8 00 00 00 eb db 48 8b 43 28 48 8d b3 c8 03 00 00 48 c7 c7 98 c6 8f 84 48 8b 10 e8 ce b3 d4 ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90
[97931.606601] RSP: 0018:ff77681d82c6be10 EFLAGS: 00010246
[97931.607089] RAX: 000000000000002d RBX: ff49cf134fbcc000 RCX: 0000000000000000
[97931.607844] RDX: 0000000000000000 RSI: ff49cf3007761540 RDI: ff49cf3007761540
[97931.608444] RBP: ffffffffc0bec3a0 R08: 0000000000000000 R09: ff77681d82c6bcb8
[97931.609042] R10: 0000000000000003 R11: ff49cf307fd3bb28 R12: ff49cf14b1de4ccc
[97931.609659] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[97931.610478] FS:  00007f2a4e061800(0000) GS:ff49cf3007740000(0000) knlGS:0000000000000000
[97931.611394] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[97931.612119] CR2: 00007f2a4e1bb620 CR3: 0000000218662006 CR4: 0000000000771ee0
[97931.613004] PKRU: 55555554
[97931.613464] Kernel panic - not syncing: Fatal exception
[97931.615489] Kernel Offset: 0x2000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[97931.616423] Rebooting in 10 seconds..

Additional Pod and ceph FS info
CEPH FS is running on 17.2.6 and I didn't see the above error anywhere else
Pods are using plain k8 functionality:

      volumeMounts:
        - name: raw-data-ro
          readOnly: true
          mountPath: /ceph-data/Raw-Data
  volumes:
    - name: raw-data-ro
      cephfs:
        monitors:
          - '10.77.50.128:6789'
          - '10.77.50.130:6789'
          - '10.77.50.132:6789'
        path: /Raw-Data
        user: oc.key
        secretRef:
          name: oc.key

Version
OKD version release: 4.13.0-0.okd-2023-09-03-082426 running on Openstack Centos9/wallaby

How reproducible
Workers reboot happens all the time - but can't be predicted

[vrutkovs]

This needs to be reported to Fedora - https://bugzilla.redhat.com/enter_bug.cgi first

[alexiusflavius]

Thanks - I'll keep this thread updated when will hear back about the bug