OKD 4.13: could not install in really airgap mode

[engineater]

OKD 4.13 do not install in "AirGap"(without the internet) mode.

4.13.0-0.okd-2023-10-28-065448

How reproducible
Prepare you own repository for pull and push OKD images.
Run this scrip in shell on Linux machine(with the internet) for pull and push OKD images from the internet.

LOCAL_SECRET_JSON='/root/pull_secret.json'
PRODUCT_REPO='openshift'
RELEASE_NAME="okd"
OCP_RELEASE='4.13.0-0.okd-2023-10-28-065448'
LOCAL_REGISTRY='nexus.local:9445'
LOCAL_REPOSITORY='repository/4.13.0-0.okd-2023-10-28-065448'
./oc adm release mirror --insecure=true -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}

After this you can see that script download everything, but did not download:

quay.io/openshift/okd-content@sha256:49dfcf845391bae2140b73ba93461acd93d75b4fd14d774e8c91c0e2acd8f0ff

In quay.io a sha256 same for this tags images:

quay.io/openshift/okd-content:4.13.0-0.okd-2023-10-28-065448-fedora-coreos
quay.io/openshift/okd-content:4.13.0-0.okd-2023-10-28-013842-fedora-coreos

if you run the booststrap node in airgap mode, in logs you can see this:

[[email protected] ~]$ journalctl
. . . . . . . . . . 
Nov 14 07:38:19 bootstrap.my.local bootstrap-pivot.sh[1889]: Pull failed. Retrying quay.io/openshift/okd-content@sha256:49dfcf845391bae2140b73ba93461acd93d75b4fd14d774e8c91c0e2acd8f0ff...
Nov 14 07:38:19 bootstrap.my.local systemd[1]: var-lib-containers-storage-overlay-e9065e7fe4871a1bc9725e03d02597f07e4d090c87d20b0b71a673a7f5452a40-merged.mount: Deactivated successfully.
Nov 14 07:38:19 bootstrap.my.local systemd[1]: var-lib-containers-storage-overlay\x2dcontainers-08358563c1bdc83b056713ba77c80f381fad41dee237d9f3571c3804eae6a94f-userdata-shm.mount: Deactivated successfully.
Nov 14 07:38:19 bootstrap.my.local systemd[1]: var-lib-containers-storage-overlay.mount: Deactivated successfully.
Nov 14 07:38:20 bootstrap.my.local bootstrap-pivot.sh[85639]: Error: initializing source docker://quay.io/openshift/okd-content@sha256:49dfcf845391bae2140b73ba93461acd93d75b4fd14d774e8c91c0e2acd8f0ff: (Mirrors also >
Nov 14 07:38:20 bootstrap.my.local bootstrap-pivot.sh[1889]: Pull failed. Retrying quay.io/openshift/okd-content@sha256:49dfcf845391bae2140b73ba93461acd93d75b4fd14d774e8c91c0e2acd8f0ff...
. . . . . . . . . . 

In the future I would like that this command "./oc adm release mirror" could pull and push quay.io/openshift/okd-content:*-fedora-coreos images too.

[vrutkovs]

Please attach log bundle