Whereabouts-reconciler image entrypoint is broken (v4.17, v4.16)

[manfuin]

Upgrade to scos-release:4.16.0-okd-scos.0 from 4.15.0-0.okd-scos-2024-01-18-223523 for me stuck on network-operator rolling out the DaemonSet "/openshift-multus/whereabouts-reconciler" (after applying fix from #1971).

The whereabouts-reconciler Pod is crashlooping with:

 [entrypoint.sh] FATAL ERROR: Unsupported OS ID=centos

Indeed, is image is based on:

cat /etc/os-release:
NAME="CentOS Stream"
VERSION="9"
ID="centos"
...

But entrypoint.sh does not handle the centos name:

# Collect host OS information
. /etc/os-release
rhelmajor=
# detect which version we're using in order to copy the proper binaries
case "${ID}" in
        rhcos|scos)
                rhelmajor=$(echo ${RHEL_VERSION} | cut -f 1 -d .)
                ;;
        rhel)
                rhelmajor=$(echo "${VERSION_ID}" | cut -f 1 -d .)
                ;;
        fedora)
                if [ "${VARIANT_ID}" == "coreos" ]; then
                        rhelmajor=8
                else
                        log "FATAL ERROR: Unsupported Fedora variant=${VARIANT_ID}"
                        exit 1
                fi
                ;;
        *) log "FATAL ERROR: Unsupported OS ID=${ID}"; exit 1
                ;;
esac

It is the same OS in scos-release:4.16.0-okd-scos.1.

But for 4.15 it was:

$ podman run -it quay.io/okd/scos-content@sha256:ee302ce6a9ac0be1e9225af2973c7be0b9db785069a376dc4ab5e12d889045d8 /bin/sh -c 'cat /etc/os-release'
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"

Any mitigation you can think of?

I tried to manually patching daemonset with dirrefent command that skips this check, but network operator is changing it back to broken one.

[manfuin]

I managed to complete the update by:

1. Excluding CNO from management:

$ oc edit clusterversion version
spec:
  overrides:
  - group: apps
    kind: Deployment
    name: network-operator
    namespace: openshift-network-operator
    unmanaged: true

2. Editing the network-operator Deployment to set WHEREABOUTS_CNI_IMAGE env variable to the working image from 4.15

Anyway I am open to suggestion what will be a proper way to apply next upgrade (assuming it will fix the issue with whereabouts). CNO definitely needs to be returned to managed state to be updated.

[manfuin]

Still an issue for 4.17.0-okd-scos.0.

The way I go forward is introducing an admission controller for openshift-multus namespace that mutates the image of whereabouts-reconciler Pods during Pod creation. This way DaemonSet remains intact and CNO can be managed again.

The working image substituted by mutation is build from upstraeam image matching the OKD release with patched entrypoint.sh:

@@ -17,7 +17,7 @@
        rhcos|scos)
                rhelmajor=$(echo ${RHEL_VERSION} | cut -f 1 -d .)
                ;;
-       rhel|centos)
+       rhel)
                rhelmajor=$(echo "${VERSION_ID}" | cut -f 1 -d .)
                ;;
        fedora)

[GingerGeek]

Thank you for the report, I'm moving this to an issue so it can be referenced by PRs etc

#2070