okd-baremetal-install uses fcos for bootstrap but rhcos for cluster

[eb4x]

Describe the bug

Using the openshift-baremetal-install the bootstrap node is running fedora coreos, but the control-plane nodes are installing redhat coreos.

I got the installer from oc adm release extract --command=openshift-baremetal-install --to ~/.local/bin quay.io/openshift/okd:4.10.0-0.okd-2022-03-07-131213

Version

openshift-baremetal-install 4.10.0-0.okd-2022-03-07-131213 (IPI)

How reproducible

100%
I'm not certain I'm doing it correctly, but even setting

platform:
  baremetal:
    clusterOSImage: "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/35.20220116.2.0/x86_64/fedora-coreos-35.20220116.2.0-metal.x86_64.raw.xz?sha256=c3778bf62397e53ce8f0ee2c02216e7438712bb9e87315288397d5283b3cece7"

didn't seem to have any effect.

[binnes]

I think this may be an issue with the text output and disk names. I've just run a test IPI install on ovirt and I see the disk images are called lab-7lgcv-rhcos and the output from the install command advises it is obtaining the RHCOS but the full output shows it is actually fetching the fedora image:

INFO Obtaining RHCOS image file from 'https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/35.20220116.2.0/x86_64/fedora-coreos-35.20220116.2.0-openstack.x86_64.qcow2.xz?sha256=dc1d6c26e6d2e8d8c3e1e50788590bd8227ca56ecfcaf565a29f98d1ca811970'

[vrutkovs]

Please attach log bundle

[eb4x]

[kni@provisioner ~]$ openshift-baremetal-install gather bootstrap --dir clusterconfigs
FATAL could not find outputs file "clusterconfigs/bootstrap.tfvars.json": stat clusterconfigs/bootstrap.tfvars.json: no such file or directory 

But I tried gathering logs from the bootstrap manually.
manual-log-bundle.tar.gz

Following this guide more or less, extracting openshift-baremetal-install from the latest okd:4.10.0-0.okd-2022-04-23-131357

openshift-baremetal-install --dir clusterconfigs --log-level debug create manifests with this install-config.yaml

---
apiVersion: v1
baseDomain: slips.pl
metadata:
  name: okd
networking:
  networkType: OVNKubernetes
compute:
- name: worker
  architecture: amd64
  replicas: 0
controlPlane:
  name: master
  platform:
    baremetal: {}
  architecture: amd64
  replicas: 3
platform:
  baremetal:
    apiVIP: 10.0.0.8
    ingressVIP: 10.0.0.9
    hosts:
    - name: master-0
      role: master
      bmc:
        username: root
        password: calvin
        address: ipmi://192.168.4.2
      bootMACAddress: c4:34:6b:b5:ad:80
      bootMode: legacy
      rootDeviceHints:
        deviceName: "/dev/sda"
      networkConfig:
        interfaces:
        - name: bond0
          state: up
          type: bond
          link-aggregation:
            mode: 802.3ad
            port:
            - eno1
            - eno2
            - eno3
            - eno4
          ipv4:
            dhcp: true
            enabled: true
        - name: bond0.10
          state: up
          type: vlan
          vlan:
            base-iface: bond0
            id: 10
          ipv4:
            dhcp: true
            enabled: true
    - name: master-1
      ...:
    - name: master-2
      ...:
pullSecret: "{\"auths\":{\"fake\":{\"auth\":\"aWQ6cGFzcwo=\"}}}"
sshKey: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFdYa5IllcvbGVs3v2cjJHnagfsBvBNTpY5gwwGe+MPu'

then openshift-baremetal-install --dir clusterconfigs --log-level debug create cluster and just sit back and wait.

Checking the bootstrap-node

ssh [email protected] -t cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Fedora CoreOS 35.20220116.2.0"

Checking on a master-* that's been pxe-booted

[core@localhost ~]$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)"

Also interesting to note, on the pxe-booted controlplane the network starts out like this;

[core@localhost ~]$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eno1             UP             
eno2             UP             
eno3             UP             
eno4             UP             
bond0            UP             172.22.0.85/24 
bond0.10@bond0   UP             10.0.0.110/8

moments later the baremetal network stops responding and this is the new state

[core@localhost ~]$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eno1             UP             
eno2             UP             
eno3             UP             
eno4             UP             
bond0            UP             172.22.0.85/24 
bond0.10@bond0   UP             10.0.0.110/8 
eno3.1000@eno3   UP             fe80::c634:6bff:feb6:a1e0/64 
eno3.10@eno3     UP             fe80::c634:6bff:feb6:a1e0/64 
eno3.3@eno3      UP             fe80::c634:6bff:feb6:a1e0/64 
eno1.1000@eno1   UP             fe80::c634:6bff:feb6:a1e0/64 
eno1.10@eno1     UP             fe80::c634:6bff:feb6:a1e0/64 
eno1.3@eno1      UP             fe80::c634:6bff:feb6:a1e0/64 
eno4.1000@eno4   UP             fe80::c634:6bff:feb6:a1e0/64 
eno4.10@eno4     UP             fe80::c634:6bff:feb6:a1e0/64 
eno4.3@eno4      UP             fe80::c634:6bff:feb6:a1e0/64 
eno2.1000@eno2   UP             fe80::c634:6bff:feb6:a1e0/64 
eno2.10@eno2     UP             fe80::c634:6bff:feb6:a1e0/64 
eno2.3@eno2      UP             fe80::c634:6bff:feb6:a1e0/64 

[vrutkovs]

Checking on a master-* that's been pxe-booted
[core@localhost ~]$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)"

Seems openshift-baremetal-installer is broken - we pass both baremetal and okd tags but don't get OKD bits there. Please create a new issue at https://bugzilla.redhat.com/enter_bug.cgi?product=OpenShift%20Container%20Platform&component=Installer

[eb4x]

https://bugzilla.redhat.com/show_bug.cgi?id=2079500

[vrutkovs]

This should be fixed in https://amd64.origin.releases.ci.openshift.org/releasetag/4.10.0-0.okd-2022-05-12-044124 - could you give it a try?

[eb4x]

oc adm release extract --command=openshift-baremetal-install --to . registry.ci.openshift.org/origin/release:4.10.0-0.okd-2022-05-12-044124

./openshift-baremetal-install version
./openshift-baremetal-install 4.10.0-0.okd-2022-05-12-044124
built from commit 2a423cb8a2e4beb108840c99ea5c895e88392183
release image registry.ci.openshift.org/origin/release@sha256:be0cc9bc5ccef062dacdfde87883a85dc4413985f685c7684eb0c068cec102e2
release architecture amd64

bootstrap: Fedora CoreOS 35
master-0: Red Hat Enterprise Linux CoreOS 410.84.202201251210-0 (Ootpa) 4.10