Error provisioning bootstrap node OKD 4.14-*

[LennertMertens]

Describe the bug

Bootstrap service failing when deploying a new user provisioned cluster on VMware vSphere (7.0.3.01700). Whenever I try to install a new cluster with any version above 4.14, the bootstrap node fails with the error reported below. When deploying any 4.13 version in the same way I don't encounter this issue.

Nov 15 07:26:19 minfra-stag-b config-render[30601]: Error: unknown flag: --rendered-manifest-files
Nov 15 07:26:19 minfra-stag-b bootkube.sh[30590]: Error: unknown flag: --rendered-manifest-files

Version

UPI installed cluster on VMware vSphere

• 4.14.0-0.okd-2023-11-14-101924
• 4.14.0-0.okd-2023-11-12-042703
• 4.14.0-0.okd-2023-10-28-073550

How reproducible

Been able to reproduce it every time when deploying the UPI installed cluster.

Log bundle

Nov 15 07:26:19 minfra-stag-b podman[30590]: 2023-11-15 07:26:19.448159447 +0000 UTC m=+0.120829128 container attach a4783197949029df48d705d2a085e60a319ae05a86a78748003a949faec591c5 (image=quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6, name=config-render, build-date=2023-07-17T10:04:14, io.openshift.build.commit.id=a9e658abf7de525b411ff005b65e061e39f58bd6, vendor=Red Hat, Inc., description=This is the base image from which all OpenShift Container Platform images inherit., io.openshift.maintainer.component=Release, io.openshift.release.operator=true, vcs-ref=a9e658abf7de525b411ff005b65e061e39f58bd6, com.redhat.component=openshift-enterprise-base-container, release=202307170916.p0.g0465934.assembly.stream, io.buildah.version=1.29.0, io.openshift.build.commit.ref=release-4.13, version=v4.13.0, distribution-scope=public, io.openshift.build.name=, com.redhat.build-host=cpt-1001.osbs.prod.upshift.rdu2.redhat.com, io.openshift.build.commit.author=, io.openshift.build.namespace=, io.openshift.build.commit.message=, io.openshift.tags=openshift,base, maintainer=Red Hat, Inc., io.openshift.build.commit.url=https://github.com/openshift/images/commit/04659348dbad0cd9b56596d5977d2934fa01d509, vcs-url=https://github.com/openshift/cluster-config-operator, vcs-type=git, io.openshift.build.source-location=https://github.com/openshift/cluster-config-operator, io.openshift.ci.from.base=sha256:76736d83edf378d282d181907c8d44df11079d67e4414c3e944952f28e5eb59f, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/agreements, io.openshift.maintainer.project=OCPBUGS, io.openshift.build.commit.date=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.13.0-202307170916.p0.g0465934.assembly.stream, name=openshift/ose-base, io.k8s.display-name=4.13-base, summary=Provides the latest release of the Red Hat Extended Life Base Image., io.k8s.description=ART equivalent image openshift-4.13-openshift-enterprise-base - rhel-8/base-repos, License=GPLv2+, io.openshift.build.source-context-dir=)
Nov 15 07:26:19 minfra-stag-b podman[30590]: 2023-11-15 07:26:19.34901306 +0000 UTC m=+0.021682766 image pull 86f8f0c0e96a34c04a15eec95ed41abbd8ccf3daa8e9ec45c05270927fe6bcb5 quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6
Nov 15 07:26:19 minfra-stag-b config-render[30601]: Error: unknown flag: --rendered-manifest-files
Nov 15 07:26:19 minfra-stag-b bootkube.sh[30590]: Error: unknown flag: --rendered-manifest-files
Nov 15 07:26:19 minfra-stag-b podman[30590]: 2023-11-15 07:26:19.480228572 +0000 UTC m=+0.152898249 container died a4783197949029df48d705d2a085e60a319ae05a86a78748003a949faec591c5 (image=quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6, name=config-render, License=GPLv2+, vcs-type=git, vcs-ref=a9e658abf7de525b411ff005b65e061e39f58bd6, io.k8s.description=ART equivalent image openshift-4.13-openshift-enterprise-base - rhel-8/base-repos, build-date=2023-07-17T10:04:14, io.openshift.build.commit.url=https://github.com/openshift/images/commit/04659348dbad0cd9b56596d5977d2934fa01d509, maintainer=Red Hat, Inc., io.buildah.version=1.29.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.13.0-202307170916.p0.g0465934.assembly.stream, io.openshift.build.name=, com.redhat.component=openshift-enterprise-base-container, io.openshift.ci.from.base=sha256:76736d83edf378d282d181907c8d44df11079d67e4414c3e944952f28e5eb59f, io.openshift.build.commit.message=, io.openshift.tags=openshift,base, com.redhat.license_terms=https://www.redhat.com/agreements, name=openshift/ose-base, io.openshift.build.commit.id=a9e658abf7de525b411ff005b65e061e39f58bd6, summary=Provides the latest release of the Red Hat Extended Life Base Image., io.openshift.maintainer.component=Release, io.openshift.build.source-context-dir=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1001.osbs.prod.upshift.rdu2.redhat.com, io.openshift.build.namespace=, io.openshift.maintainer.project=OCPBUGS, io.openshift.build.commit.ref=release-4.13, io.openshift.release.operator=true, io.openshift.build.commit.author=, io.k8s.display-name=4.13-base, io.openshift.expose-services=, architecture=x86_64, release=202307170916.p0.g0465934.assembly.stream, description=This is the base image from which all OpenShift Container Platform images inherit., version=v4.13.0, distribution-scope=public, io.openshift.build.source-location=https://github.com/openshift/cluster-config-operator, vcs-url=https://github.com/openshift/cluster-config-operator, io.openshift.build.commit.date=)
Nov 15 07:26:19 minfra-stag-b podman[30611]: 2023-11-15 07:26:19.531882059 +0000 UTC m=+0.043112508 container remove a4783197949029df48d705d2a085e60a319ae05a86a78748003a949faec591c5 (image=quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6, name=config-render, io.openshift.build.commit.id=a9e658abf7de525b411ff005b65e061e39f58bd6, io.openshift.build.source-location=https://github.com/openshift/cluster-config-operator, License=GPLv2+, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/agreements, release=202307170916.p0.g0465934.assembly.stream, io.openshift.build.namespace=, io.k8s.description=ART equivalent image openshift-4.13-openshift-enterprise-base - rhel-8/base-repos, io.openshift.expose-services=, vendor=Red Hat, Inc., name=openshift/ose-base, io.openshift.build.name=, vcs-url=https://github.com/openshift/cluster-config-operator, io.openshift.build.commit.message=, vcs-ref=a9e658abf7de525b411ff005b65e061e39f58bd6, io.openshift.build.commit.url=https://github.com/openshift/images/commit/04659348dbad0cd9b56596d5977d2934fa01d509, io.openshift.tags=openshift,base, io.buildah.version=1.29.0, build-date=2023-07-17T10:04:14, description=This is the base image from which all OpenShift Container Platform images inherit., distribution-scope=public, summary=Provides the latest release of the Red Hat Extended Life Base Image., io.openshift.build.commit.ref=release-4.13, io.openshift.ci.from.base=sha256:76736d83edf378d282d181907c8d44df11079d67e4414c3e944952f28e5eb59f, version=v4.13.0, com.redhat.build-host=cpt-1001.osbs.prod.upshift.rdu2.redhat.com, maintainer=Red Hat, Inc., io.openshift.build.commit.author=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.13.0-202307170916.p0.g0465934.assembly.stream, io.openshift.maintainer.project=OCPBUGS, io.openshift.build.commit.date=, io.openshift.maintainer.component=Release, io.openshift.release.operator=true, io.openshift.build.source-context-dir=, architecture=x86_64, com.redhat.component=openshift-enterprise-base-container, io.k8s.display-name=4.13-base)
Nov 15 07:26:19 minfra-stag-b systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 07:26:19 minfra-stag-b systemd[1]: bootkube.service: Failed with result 'exit-code'.
Nov 15 07:26:19 minfra-stag-b systemd[1]: bootkube.service: Consumed 2.708s CPU time.

[vrutkovs]

Please attach log bundle

[LennertMertens]

I recreared the cluster so unfortunately I'm unable to create the log bundle. I downgraded to 4.13 and then did an update to 4.14. That did work.