Process (conmon) of user 0 dumped core.

[hunter86bg]

Describe the bug
Latest baremetal fails to bootstrap. Conmon process coredumps.

Version
fedora-coreos-35.20211203.3.0-metal.x86_64.raw.xz
openshift-client-linux-4.9.0-0.okd-2021-12-12-025847.tar.gz
openshift-install-linux-4.9.0-0.okd-2021-12-12-025847.tar.gz

How reproducible
100% . Retried several times before managing to reach the master nodes and check the logs.

Log bundle

Dec 29 20:34:17 okd4-control-plane-2 systemd-coredump[1694]: [🡕] Process 1612 (conmon) of user 0 dumped core.
                                                             
                                                             Found module linux-vdso.so.1 with build-id: 3a757ae982872004a262a28ed5528bfb98723035
                                                             Found module libgpg-error.so.0 with build-id: 081975d0a3416374b4883b2f1639fd3c9df42390
                                                             Found module ld-linux-x86-64.so.2 with build-id: da6dc8cd9ecc65cbd54815426326bae9614f727c
                                                             Found module libgcc_s.so.1 with build-id: 279c1648ee67377b5de1c45a244740beea48423e
                                                             Found module libgcrypt.so.20 with build-id: 711d41580c5a8649a79a8430a985dac3e25b5ba2
                                                             Found module libcap.so.2 with build-id: 0214aa9cc6a8646eb9ec27ab7dda6a2219da6500
                                                             Found module liblz4.so.1 with build-id: fd02c4542a1ce1ad6c503b958447be1c37a4afee
                                                             Found module libzstd.so.1 with build-id: c594c701d8f5ba174a5be306973fa9803d65e327
                                                             Found module liblzma.so.5 with build-id: 7fec53ce7cba9489f130c99cfd3ace82e9dde0ee
                                                             Found module libpcre.so.1 with build-id: fcbdb4731e50f20a47b62341a6841dd616fe495d
                                                             Found module libc.so.6 with build-id: 148792ddec60e0b15e20655fdc10c35a5c394413
                                                             Found module libseccomp.so.2 with build-id: d9e37e30e12bcd8de6d550fda9c874c8870067d9
                                                             Found module libsystemd.so.0 with build-id: e4996d7fd9705035bdd6072e68c9dc65647d7d1c
                                                             Found module libglib-2.0.so.0 with build-id: 2b899125cfc5dc91a26a9aef52959c6da0a5445b
                                                             Found module conmon with build-id: d185e6305296ab5c3539014320a9f083e49b214c
                                                             Stack trace of thread 1612:
                                                             #0  0x00007f22b02b51ba __memchr_sse2 (libc.so.6 + 0xaa1ba)
                                                             #1  0x00007f22b0494585 sd_journal_sendv (libsystemd.so.0 + 0x5f585)
                                                             #2  0x00007f22b0496061 sd_journal_sendv_with_location (libsystemd.so.0 + 0x61061)
                                                             #3  0x0000000000405e10 write_journald (conmon + 0x5e10)
                                                             #4  0x0000000000406364 write_to_logs (conmon + 0x6364)
                                                             #5  0x00000000004093e9 read_stdio (conmon + 0x93e9)
                                                             #6  0x0000000000409445 stdio_cb (conmon + 0x9445)
                                                             #7  0x00007f22b055c33f g_main_context_dispatch (libglib-2.0.so.0 + 0x5533f)
                                                             #8  0x00007f22b05b12b8 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa2b8)
                                                             #9  0x00007f22b055b903 g_main_loop_run (libglib-2.0.so.0 + 0x54903)
                                                             #10 0x0000000000404d9a main (conmon + 0x4d9a)
                                                             #11 0x00007f22b0238560 __libc_start_call_main (libc.so.6 + 0x2d560)
                                                             #12 0x00007f22b023860c __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d60c)
                                                             #13 0x0000000000405235 _start (conmon + 0x5235)
                 
                                                             Stack trace of thread 1614:
                                                             #0  0x00007f22b031081f __poll (libc.so.6 + 0x10581f)
                                                             #1  0x00007f22b05b124c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa24c)
                                                             #2  0x00007f22b05599e3 g_main_context_iteration (libglib-2.0.so.0 + 0x529e3)
                                                             #3  0x00007f22b0559a31 glib_worker_main (libglib-2.0.so.0 + 0x52a31)
                                                             #4  0x00007f22b058b122 g_thread_proxy (libglib-2.0.so.0 + 0x84122)
                                                             #5  0x00007f22b0298b17 start_thread (libc.so.6 + 0x8db17)
                                                             #6  0x00007f22b031d6c0 __clone3 (libc.so.6 + 0x1126c0)
░░ Subject: Process 1612 (conmon) dumped core
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ Documentation: man:core(5)
░░ 
░░ Process 1612 (conmon) crashed and dumped core.
░░ 
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.

core.conmon.0.f1a6f387b0ca47849f000f9297898cad.1612.1640810057000000.zst.gz

[vrutkovs]

fedora-coreos-35.20211203.3.0-metal.x86_64.raw.xz

is it working with https://github.com/openshift/okd/blob/master/FAQ.md#which-fedora-coreos-should-i-use?

[hunter86bg]

Yep, 34 works fine

[bostrt]

I ran into this issue installing OKD 4.9 (4.9.0-0.okd-2022-02-12-140851) on FCOS 35 (fedora-coreos-35.20220131.3.0-digitalocean.x86_64.qcow2.gz) on Digital Ocean. On my failing control plane node, the segfaulting conmon process is associated with machine-config-daemon-pull.service. If I run check the exited machine config container logs, there's a large amount of unreadable binary displayed which may suggest why conmon's journald write call is failing?

To be clear, using fcos 34 works fine for me.

[bostrt]

xref openshift/machine-config-operator#2925