Please document in a "known issues" section that installing OKD v4.16/v4.17 in UPI+ABI/AI on baremetal fails

[titou10titou10]

So OKD v4.16/4.17 is officially out: https://github.com/okd-project/okd/releases/tag/4.17.0-okd-scos.0
Good work!

Could you add somewhere that installing those version in UPI with the agent based installer (ABI) or agent installer (AI) on baremetal does not work .
The problem and possible workaround is described here and in many other discussions (#2035, #2015, #1983 ...)

@JaimeMagiera suggested this in a comment here: #2019 (reply in thread)

IMHO it could spare some time for people that use this method to install OKD in UPI with ABI/AI on baremetal

[GingerGeek]

I've created okd-project/okd-web#47 and will read through these discussions and form an additional page for the update notes section. If you have time to contribute and could write something yourself please let me know!

[titou10titou10]

OK
I'll try to write something.
If the issues are the same as before with this release, it will be the synthesis of #2015 (FCOS bootstrap "read-only") and #2035 (issue with PostgreSQL used by the agent installer not starting..)
I'll install v4.17 only, not v4.16 . I will "move" the workload from v4.15 to v4.17, taking the opportunity to start a fresh new cluster.
It is written elsewhere that v4.16 is not meant to be installed "straight", but only used as a migration step from v4.15 to v4.17, so it is useless to document v4.16 ABI/AI UPI baremetal install problems

[GingerGeek]

Yes, thank you. I have seen your response on the issue and will get that up shortly.

Correct that we don't see reasons why 4.16 would need to be freshly installed as it's just pass-thru from existing 4.15 clusters. New clusters should be 4.17.

Thank you for the contributions :)

[CrAazZyMaN21]

Hi,

I got it working on installing OKD 4.17 from stratch with PXEBoot and Ignition Files on VMs running Proxmox. I used an older SCOS Live ISO (scos-413.9.202303020610-0-live.x86_64.iso) metioned in the okd-scos repo (okd-project/okd-scos#11) where i extraced the vmlinuz, initrd.img and rootfs.img. With those files I cloud PXEBoot the boostrap, master and worker Ignition files created with openshift-install and install the OKD Cluster without any issues at all. This fixed all the issues I had previously with either updating or installing the OKD Cluster from scratch with other workarounds mentioned in the other discussions. When the offical SCOS Images will be released the install shouldn't be a problem anymore

I hope this helps others trying to setup a OKD Cluster from scratch 😄

Thanks for the OKD Project ❤️

[titou10titou10]

I got it working on installing OKD 4.17 from stratch with PXEBoot and Ignition Files on VMs running Proxmox.

I'm confused here. What do you mean by "from stratch with PXEBoot" ? do you use a dedicated bootstrap node and n http server for the iPXE/ign files? if so this is not an ABI/AI where there is no dedicated node ...

[CrAazZyMaN21]

Oh sry, my bad. I answered to the wrong thread. Yeah I used a dedicated bootstrap node and a http server with ipxe and ign files.

[holbjo]

Experiencing a somewhat similar issue here with 4.17.0-okd-scos.1

$ openshift-install version
openshift-install 4.17.0-okd-scos.1
built from commit baa59e5213e017d66a818c975f6093b1db697dda
release image quay.io/okd/scos-release@sha256:0a8e7694df4234b0d0987a266e373640354b997f0522dcdee902019304165908
release architecture multi
default architecture amd64

$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.qemu.formats."qcow2.xz".disk.location'
https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/x86_64/fedora-coreos-39.20231101.3.0-qemu.x86_64.qcow2.xz

Although not seeing the specific log entry for the failed PSQL lock as referenced at 2015 I do see:

cla-sno service[97401]: time="2025-01-04T04:25:56Z" level=info msg="Failed to connect to DB, retrying" func=main.setupDB.func1 file="/src/cmd/main.go:695" error="failed to connect to `host=127.0.0.1 user=admin database=installer`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)"

After applying the changes to the assisted-service-db.service the process continues and ends in a loop of release-image-pivot.service and kublet.service

cla-sno bootkube.sh[290512]: /usr/local/bin/bootkube.sh: line 85: oc: command not found
cla-sno systemd[1]: bootkube.service: Main process exited, code=exited, status=127/n/a
cla-sno systemd[1]: bootkube.service: Failed with result 'exit-code'

[root@cla-sno ~]# systemctl status  kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: activating (auto-restart) (Result: exit-code) since Sat 2025-01-04 05:34:13 UTC; 6s ago
    Process: 614208 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
    Process: 614210 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/kubelet-plugins/volume/exec (code=exited, status=0/SUCCESS)
    Process: 614213 ExecStartPre=/usr/local/bin/kubelet-pause-image.sh (code=exited, status=0/SUCCESS)
    Process: 614302 ExecStart=/usr/local/bin/kubelet.sh (code=exited, status=127)
   Main PID: 614302 (code=exited, status=127)
        CPU: 462ms

[root@cla-sno ~]# systemctl status release-image-pivot
× release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image
     Loaded: loaded (/etc/systemd/system/release-image-pivot.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Sat 2025-01-04 05:22:55 UTC; 6s ago
    Process: 545833 ExecStart=/usr/local/bin/bootstrap-pivot.sh (code=exited, status=1/FAILURE)
   Main PID: 545833 (code=exited, status=1/FAILURE)
        CPU: 691ms

Jan 04 05:22:52 cla-sno podman[545856]: 2025-01-04 05:22:52.616537376 +0000 UTC m=+0.118252734 container died 535826bada912069a810f2b58bbcb9fe457e7325c3f96>
Jan 04 05:22:52 cla-sno podman[545856]: 2025-01-04 05:22:52.520807597 +0000 UTC m=+0.022522855 image pull 96183e62b61764a0cd096cd9e1f1e7b3dd609b7f44fea1b22>
Jan 04 05:22:52 cla-sno podman[545887]: 2025-01-04 05:22:52.659901535 +0000 UTC m=+0.033312068 container remove 535826bada912069a810f2b58bbcb9fe457e7325c3f>
Jan 04 05:22:52 cla-sno bootstrap-pivot.sh[545833]: Pulling quay.io/okd/scos-content@sha256:4027bd5d28bd5ee9fb655909069ce145af5b0e63544f99dc8410f8df4f627db>
Jan 04 05:22:55 cla-sno bootstrap-pivot.sh[545911]: 28c0f5216da0ad68592b542d100daaa6c848ebad0f7451858806d762b01c2a24
Jan 04 05:22:55 cla-sno podman[545911]: 2025-01-04 05:22:55.024641192 +0000 UTC m=+2.305642642 image pull 28c0f5216da0ad68592b542d100daaa6c848ebad0f7451858>
Jan 04 05:22:55 cla-sno bootstrap-pivot.sh[545962]: error: Remounting /sysroot read-write: Permission denied
Jan 04 05:22:55 cla-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 05:22:55 cla-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
Jan 04 05:22:55 cla-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.

[titou10titou10]

@holbjo you've hit the 2 current problems on installing OKD v4.16/17 in UPI with ABI/AI.

Check okd-project/okd-web#47 (comment)

The logs related to the Postgresql startup problem now does not directly appears in the systemd service or journalctl but in the logs of one of the pods started again and again by it.
If you are fast enough you can capture the logs of the problem withpodman logs...

[UPDATE]
To see what's going wrong with the assisted-service-dbservice, manually run the command from the/etc/systemd/system/assisted-service-dbservice:

  /usr/bin/podman run --net host --user=postgres --name=assisted-db2 --env-file=/usr/local/share/assisted-service/assisted-db.env --replace -it $SERVICE_IMAGE ./start_db.sh
  > FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": No such file or directory"