[OKD-SCOS v4.16] UPI install with agent installer or assisted installer fails for 4.16.0-0.okd-scos-2024-08-01-132038

[titou10titou10]

Context

Trying to install a SNO (Single Node) cluster:

• with OKD-SCOS v4.16.0-0.okd-scos-2024-08-01-132038 ("4-scos-next" version)
• on baremetal with UPI
• using agent based installer (ABI) or Assisted Installer (see below)
• on Proxmox (qemu) vm: 40G RAM, 100G SSD, 10 vCPUS

It is important to note that the install works perfectly well with the exact same agent and install config files for

• OKD-SCOS v4.15.0-0.okd-scos-2024-01-18-223523 ("stable")
• OKD-FCOS v4.15.0-0.okd-2024-03-10-010116 ("stable")

I also have tried for a multi-node cluster, it fails the same way

Summary

It fails with the following error from the "release-image-pivot" service:

okd-sno bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied

Details

install-config.yaml:

  apiVersion: v1
  baseDomain: denis.prive
  metadata:
    name: okd-sno
  controlPlane:
    architecture: amd64
    hyperthreading: Enabled   
    name: master 
    replicas: 1
  compute:
    - architecture: amd64
      hyperthreading: Enabled   
      name: worker
      replicas: 0 
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
    - cidr: 10.132.0.0/16
      hostPrefix: 23 
    machineNetwork:
    - cidr: 192.168.5.0/24
    serviceNetwork: 
    - 172.31.0.0/16
  platform:
    none: {} 
  fips: false 
  sshKey: 'ssh-rsa AAAA{...}' 
  pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"b3{...}}}}' 
  additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    {...}
    -----END CERTIFICATE-----

agent-config.yaml

  apiVersion: v1beta1
  kind: AgentConfig
  metadata:
    name: okd-sno
  rendezvousIP: 192.168.5.71
  additionalNTPSources:
  - time.nrc.ca
  - time.google.com
  - time.cloudflare.com
  hosts: 
    - hostname: okd-sno
      role: master
      interfaces:
        - name: ens18
          macAddress: xx:xx:xx:xx:xx:71
      rootDeviceHints: 
        deviceName: /dev/sda
      networkConfig: 
        interfaces:
          - name: ens18
            type: ethernet
            state: up
            mac-address: xx:xx:xx:xx:xx:71
            ipv4:
              enabled: true
              dhcp: true
              #auto-dns: true
              auto-dns: false
              auto-gateway: true
              auto-routes: true
            ipv6:
              enabled: false
        dns-resolver:
          config:
            search:
              - denis.prive
            server:
              - 192.168.5.18
              - 8.8.8.8

The install fails after a few minutes the node boots, but the process fails, looping forever

ssh core@okd-sno
sudo su -

"kubelet" service:

journalctl -u kubelet

Aug 18 14:27:53 okd-sno systemd[1]: Starting kubelet.service - Kubernetes Kubelet...
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.367956979 +0000 UTC m=+0.181056791 container create 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.31278308 +0000 UTC m=+0.125882875 image pull 60a77a109240508f118aed8a08c7ffe822c6695f21b7761a9dbe8e008a9d54d2 quay.io/okd>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.593446922 +0000 UTC m=+0.406546764 container init 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (image>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.603769018 +0000 UTC m=+0.416868833 container start 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (imag>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.606413875 +0000 UTC m=+0.419513627 container attach 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:54 okd-sno inspiring_raman[8518]: quay.io/okd/scos-content@sha256:8d99668e00b1c44a559b7869f660484e58071c581608669ef71c908499c82fc2
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.700681983 +0000 UTC m=+0.513781795 container died 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (image>
Aug 18 14:27:54 okd-sno podman[8532]: 2024-08-18 14:27:54.879290798 +0000 UTC m=+0.151817502 container remove 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:55 okd-sno kubelet.sh[8566]: /usr/local/bin/kubelet.sh: line 7: /usr/bin/kubelet: No such file or directory
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Main process exited, code=exited, status=127/n/a
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 18 14:27:55 okd-sno systemd[1]: Failed to start kubelet.service - Kubernetes Kubelet.
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Consumed 1.945s CPU time.
Aug 18 14:28:05 okd-sno systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3.
Aug 18 14:28:08 okd-sno systemd[1]: Starting kubelet.service - Kubernetes Kubelet...

systemctl status  kubelet

? kubelet.service - Kubernetes Kubelet
    Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
   Drop-In: /usr/lib/systemd/system/service.d
            +-10-timeout-abort.conf
    Active: activating (auto-restart) (Result: exit-code) since Sun 2024-08-18 14:33:32 UTC; 3s ago
   Process: 27054 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
   Process: 27055 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/kubelet-plugins/volume/exec (code=exited, status=0/SUCCESS)
   Process: 27058 ExecStartPre=/usr/local/bin/kubelet-pause-image.sh (code=exited, status=0/SUCCESS)
   Process: 27135 ExecStart=/usr/local/bin/kubelet.sh (code=exited, status=127)
  Main PID: 27135 (code=exited, status=127)
       CPU: 1.587s

"release-image-pivot" service:

journalctl -u  release-image-pivot

Aug 18 14:31:57 okd-sno systemd[1]: Starting release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image...
Aug 18 14:31:57 okd-sno bootstrap-pivot.sh[22255]: touch: cannot touch '/usr/.test': Read-only file system
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.315765956 +0000 UTC m=+0.156637380 container create afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (im>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.26111923 +0000 UTC m=+0.101990587 image pull 60a77a109240508f118aed8a08c7ffe822c6695f21b7761a9dbe8e008a9d54d2 quay.io/ok>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.506675673 +0000 UTC m=+0.347546962 container init afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.510429857 +0000 UTC m=+0.351301098 container start afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (ima>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.511434609 +0000 UTC m=+0.352305922 container attach afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (im>
Aug 18 14:31:58 okd-sno gracious_ride[22350]: quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.614285575 +0000 UTC m=+0.455156810 container died afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno podman[22388]: 2024-08-18 14:31:58.7502221 +0000 UTC m=+0.120921124 container remove afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno bootstrap-pivot.sh[22249]: Pulling quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b...
Aug 18 14:31:59 okd-sno bootstrap-pivot.sh[22412]: 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c
Aug 18 14:31:59 okd-sno podman[22412]: 2024-08-18 14:31:59.897322251 +0000 UTC m=+0.930918869 image pull 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c quay.io/o>
Aug 18 14:32:00 okd-sno bootstrap-pivot.sh[22459]: error: Remounting /sysroot read-write: Permission denied
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
Aug 18 14:32:00 okd-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Consumed 2.767s CPU time.
Aug 18 14:32:03 okd-sno systemd[1]: Starting release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image...

systemctl status release-image-pivot

× release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image
     Loaded: loaded (/etc/systemd/system/release-image-pivot.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             +-10-timeout-abort.conf
     Active: failed (Result: exit-code) since Sun 2024-08-18 14:33:06 UTC; 2s ago
    Process: 25649 ExecStart=/usr/local/bin/bootstrap-pivot.sh (code=exited, status=1/FAILURE)
   Main PID: 25649 (code=exited, status=1/FAILURE)
        CPU: 2.237s

 Aug 18 14:33:04 okd-sno podman[25675]: 2024-08-18 14:33:04.192809038 +0000 UTC m=+0.384116252 container died ed9e60ddea89f2f661318693f279a1de1278f60e62af95feb7d1e3d9b44b6f02 (imag>
 Aug 18 14:33:04 okd-sno podman[25707]: 2024-08-18 14:33:04.415531285 +0000 UTC m=+0.184837394 container remove ed9e60ddea89f2f661318693f279a1de1278f60e62af95feb7d1e3d9b44b6f02 (im>
 Aug 18 14:33:04 okd-sno bootstrap-pivot.sh[25649]: Pulling quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b...
 Aug 18 14:33:05 okd-sno bootstrap-pivot.sh[25740]: 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c
 Aug 18 14:33:05 okd-sno podman[25740]: 2024-08-18 14:33:05.418204678 +0000 UTC m=+0.805573537 image pull 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c quay.io/o>
 Aug 18 14:33:05 okd-sno bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
 Aug 18 14:33:06 okd-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Consumed 2.237s CPU time.

So the "release-image-pivot" fails to start because this problem?:

bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied

Other (pertinent?) info:

approve-csr.service

systemctl status approve-csr.service
● approve-csr.service - Approve CSRs during bootstrap phase
     Loaded: loaded (/etc/systemd/system/approve-csr.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sun 2024-08-18 14:27:27 UTC; 48min ago
   Main PID: 7048 (bash)
      Tasks: 2 (limit: 47999)
     Memory: 652.0K
        CPU: 4.558s
     CGroup: /system.slice/approve-csr.service
             ├─  7048 bash /usr/local/bin/approve-csr.sh /opt/openshift/auth/kubeconfig-loopback
             └─162821 sleep 20

Aug 18 15:12:30 okd-sno approve-csr.sh[152879]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:12:50 okd-sno approve-csr.sh[153717]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:10 okd-sno approve-csr.sh[154652]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:30 okd-sno approve-csr.sh[155904]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:50 okd-sno approve-csr.sh[156695]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:10 okd-sno approve-csr.sh[158162]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:30 okd-sno approve-csr.sh[159034]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:50 okd-sno approve-csr.sh[160435]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:15:10 okd-sno approve-csr.sh[161571]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:15:30 okd-sno approve-csr.sh[162817]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found

podman

podman ps
CONTAINER ID  IMAGE                                                                                             COMMAND               CREATED         STATUS         PORTS       NAMES
7f0dce9bf83f  localhost/podman-pause:4.7.0-1695838680                                                                                 45 minutes ago  Up 45 minutes              691c75a59612-infra
4553e7249821  quay.io/okd/scos-content@sha256:9277ace8253e579f8652eedb149d665287ae5dde4c3ba129c4ff70774bee4ac6  /bin/bash start_d...  44 minutes ago  Up 44 minutes              assisted-db
3279197317a9  quay.io/okd/scos-content@sha256:9277ace8253e579f8652eedb149d665287ae5dde4c3ba129c4ff70774bee4ac6  /assisted-service     44 minutes ago  Up 44 minutes              service
4d7f336db99a  quay.io/okd/scos-content@sha256:32818d2c26e3a11dc7acc313f8d277a3d11602675461af627502d32f919f93a1  next_step_runner ...  44 minutes ago  Up 44 minutes              next-step-runner
25ec9c2bee7a  quay.io/okd/scos-content@sha256:f78e6f0722a5a6a89423046e1abc68bd237551a9d93dfeac0e92e4ea34cb50bb  --role bootstrap ...  41 minutes ago  Up 41 minutes              assisted-installer

podman logs 4d7f336db99a
WARN[0000] Agent Authentication Token not set

Message from the installer:

  ./openshift-install --dir install agent wait-for bootstrap-complete --log-level=debug

  DEBUG OpenShift Installer 4.16.0-0.okd-scos-2024-08-01-132038
  DEBUG Built from commit ade29f4fbb06f3e9059e635dab4409304d70b38d
  DEBUG asset directory: install
  {...}
  INFO Unable to retrieve cluster metadata from Agent Rest API: no clusterID known for the cluster
  INFO Cluster is not ready for install. Check validations
  INFO Registered infra env
  INFO Updated image information (Image type is "full-iso", SSH public key is set)
  DEBUG Cluster validation: All hosts in the cluster are ready to install.
  WARNING Cluster validation: Single-node clusters must have a single control plane node and no workers.
  DEBUG Cluster validation: API virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: API virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: The Cluster Network CIDR is defined.
  DEBUG Cluster validation: The base domain is defined.
  DEBUG Cluster validation: Ingress virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: Ingress virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: The Machine Network CIDR is defined.
  DEBUG Cluster validation: The Cluster Machine CIDR is not required: User Managed Networking
  DEBUG Cluster validation: The Cluster Network prefix is valid.
  DEBUG Cluster validation: The cluster has a valid network type
  DEBUG Cluster validation: Same address families for all networks.
  DEBUG Cluster validation: No CIDRS are overlapping.
  DEBUG Cluster validation: No ntp problems found
  DEBUG Cluster validation: The Service Network CIDR is defined.
  DEBUG Cluster validation: cnv is disabled
  DEBUG Cluster validation: lso is disabled
  DEBUG Cluster validation: lvm is disabled
  DEBUG Cluster validation: mce is disabled
  DEBUG Cluster validation: odf is disabled
  DEBUG Cluster validation: Platform requirements satisfied
  DEBUG Cluster validation: The pull secret is set.
  INFO Host 33f1ba18-536d-4d4c-bd58-f669b5f90b97: Successfully registered
  DEBUG Host okd-sno validation: Valid inventory exists for the host
  DEBUG Host okd-sno validation: Sufficient CPU cores
  DEBUG Host okd-sno validation: Sufficient minimum RAM
  DEBUG Host okd-sno validation: Sufficient disk capacity
  DEBUG Host okd-sno validation: Sufficient CPU cores for role master
  DEBUG Host okd-sno validation: Sufficient RAM for role master
  DEBUG Host okd-sno validation: Hostname okd-sno is unique in cluster
  DEBUG Host okd-sno validation: Hostname okd-sno is allowed
  DEBUG Host okd-sno validation: Speed of installation disk has not yet been measured
  DEBUG Host okd-sno validation: Host is compatible with cluster platform none
  DEBUG Host okd-sno validation: Host agent is compatible with the service
  DEBUG Host okd-sno validation: No request to skip formatting of the installation disk
  DEBUG Host okd-sno validation: All disks that have skipped formatting are present in the host inventory
  DEBUG Host okd-sno validation: Host is connected
  DEBUG Host okd-sno validation: Media device is connected
  DEBUG Host okd-sno validation: No Machine Network CIDR needed: User Managed Networking
  DEBUG Host okd-sno validation: Host belongs to all machine network CIDRs
  DEBUG Host okd-sno validation: Host has connectivity to the majority of hosts in the cluster
  DEBUG Host okd-sno validation: Platform Standard PC (i440FX + PIIX, 1996) is allowed
  WARNING Host okd-sno validation: Host couldn't synchronize with any NTP server
  DEBUG Host okd-sno validation: Host clock is synchronized with service
  DEBUG Host okd-sno validation: All required container images were either pulled successfully or no attempt was made to pull them
  DEBUG Host okd-sno validation: Network latency requirement has been satisfied.
  DEBUG Host okd-sno validation: Packet loss requirement has been satisfied.
  DEBUG Host okd-sno validation: Host has been configured with at least one default route.
  DEBUG Host okd-sno validation: Domain name resolution for the api.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Domain name resolution for the api-int.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Domain name resolution for the *.apps.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Host subnets are not overlapping
  DEBUG Host okd-sno validation: No IP collisions were detected by host 33f1ba18-536d-4d4c-bd58-f669b5f90b97
  DEBUG Host okd-sno validation: cnv is disabled
  DEBUG Host okd-sno validation: lso is disabled
  DEBUG Host okd-sno validation: lvm is disabled
  DEBUG Host okd-sno validation: mce is disabled
  DEBUG Host okd-sno validation: odf is disabled
  WARNING Host okd-sno: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server)
  WARNING Cluster validation: The cluster has hosts that are not ready to install.
  INFO Cluster validation: The cluster has the exact amount of dedicated control plane nodes.
  INFO Host okd-sno: updated status from insufficient to known (Host is ready to be installed)
  INFO Cluster is ready for install
  INFO Cluster validation: All hosts in the cluster are ready to install.
  INFO Preparing cluster for installation
  INFO Host okd-sno: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
  INFO Host okd-sno validation: Host NTP is synced
  INFO Host okd-sno: validation 'ntp-synced' is now fixed
  INFO Host okd-sno: New image status quay.io/okd/scos-content@sha256:f78e6f0722a5a6a89423046e1abc68bd237551a9d93dfeac0e92e4ea34cb50bb. result: success. time: 5.11 seconds; size: 332.66 Megabytes; download rate: 68.29 MBps
  INFO Host okd-sno: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
  INFO Cluster installation in progress
  INFO Host okd-sno: updated status from preparing-successful to installing (Installation is in progress)
  INFO Host: okd-sno, reached installation stage Installing: bootstrap
  INFO Host: okd-sno, reached installation stage Waiting for bootkube

[madpearl]

I've just checked that, agent-based installer is not a part of docs.okd.io, you can find it on docs.openshift.com. May be this method is not supported by OKD!

[titou10titou10]

Follow-up

Also fails with the Assisted Installer:

The installation fails with the Assisted Installer in the same way. Not a big surprise as the assisted installer uses the agent installer under the hood
I use the technique described here with

RELEASE_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038","version":"4.16.0-0.okd-scos-2024-08-01-132038","default":true,"support_level":"beta"}]'

Workaround for a success with Assisted Installer:

The installation works when replacing the embedded FCOS bootstrap OS by RHCOS:

OS_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","url":"https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/4.16.3/rhcos-4.16.3-x86_64-live.x86_64.iso","version":"416.94.202406251923-0"}]'
RELEASE_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038","version":"4.16.0-0.okd-scos-2024-08-01-132038","default":true,"support_level":"beta"}]'

Workaround for a success with Agent Installer (ABI):

Overriding the bootstrap OS image with a RHCOS image make the installation a success also when installing OKD via ABI by using the following when building the install image:

export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/4.16.3/rhcos-4.16.3-x86_64-live.x86_64.iso
oc adm release extract --command=openshift-install quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038
./openshift-install agent create image --dir install --log-level=debug

I did not choose a random OS bootstrap OS image, this is the one for v4.16 specified for an OCP installation via the ABI as specified here: https://github.com/openshift/assisted-service/blob/d3324b06a7c7772f4619c3ab13dd8c0706e55fd9/deploy/podman/configmap.yml#L25

Q:

• is it planned for the next version of OKD-SCOS to include an SCOS/RHCOS bootstrap image instead of the FCOS OS image?

[arichard42]

Hi,

I have encountered the exact same issue.

Overriding the bootstrap OS image with a RHCOS image as proposed is not working for me as my lab hardware is using an LSI hardware (Dell Perc H310 using an IT firmware) that is not recognize by the RHCOS image (the mpt3sas driver on RHCOS has disabled the support for this driver).

Also the other alternative to use openshift-install-linux-4.15.0-0.okd-scos-2024-01-18-223523 do not work for me : the rendezvous host fails to start the bootstrap with an error 'pull secret for new cluster is invalid: pull secret must contain auth for "registry.ci.openshift.org"'.

@titou10titou10, shouldn't you open an issue about this problem ?

Best Regards,

Alain

[akiselev1]

I reproduced this problem with 4.16.0-0.okd-scos-2024-09-27-110344 image on Dell PowerEdge R740.
The error message looks slightly different now:
okd-sno bootstrap-pivot.sh[2073771]: error: Read-only file system
But the rest is exactly the same.

I also verified the proposed RHCOS workaround. That worked fine for me.