registry of working IPI deployments

[pzi123]

I have a general question and suggestion: why is it so quiet here? This google group okd-wg and github.com/okd-project/okd/discussions are the only communication channels on OKD. It is a very complex system with tens of installation flavors on tens of platforms. Releases are generated every few days with the latest updated often. On okd-wg I count 7 posts a month and on github some 10 discussions a month. I tried to look up the slack channel #openshift-users but it is reserved, on matrix.org chat we have just build notifications. I regularly build IPI vsphere clusters and since OKD 4.8 release I fail to stand up one. Posting failures is very difficult as there is so many moving parts. Would it make sense to start a registry of working install-config.yaml and description of the versions of the platforms used? At least at that point if someone says a particular release X is working on platform Y then it will tell us that we are doing something wrong.

[vrutkovs]

I regularly build IPI vsphere clusters and since OKD 4.8 release I fail to stand up one

Please create a discussion for a particular case and link to a log bundle. This is a common case and most of the posted one are now solved.

Would it make sense to start a registry of working install-config.yaml and description of the versions of the platforms used?

The majority of these are baremetal installations, where install-config doesn't include most critical details (usually networking). So a "working install-config" won't be useful to anybody else.

At least at that point if someone says a particular release X is working on platform Y

We have automated tests which ensure each release works on most of the supported platforms (AWS, GCP and baremetal-via-assisted-installer). The only notable exceptions are Azure and arm64 (see related issues).

[pzi123]

I regularly build IPI vsphere clusters and since OKD 4.8 release I fail to stand up one

Please create a discussion for a particular case and link to a log bundle. This is a common case and most of the posted one are now solved.

will do, thanks

Would it make sense to start a registry of working install-config.yaml and description of the versions of the platforms used?

The majority of these are baremetal installations, where install-config doesn't include most critical details (usually networking). So a "working install-config" won't be useful to anybody else.

My experience with install-config.yaml with IPI on vsphere is that I basically reuse them verbatim from build to build. The networking may change in machineNetwork apiVIP ingressVIP but that is to be expected and easy to do.

At least at that point if someone says a particular release X is working on platform Y

We have automated tests which ensure each release works on most of the supported platforms (AWS, GCP and baremetal-via-assisted-installer). The only notable exceptions are Azure and arm64 (see related issues).

Could you share the process to check the results of a specific build and test that was run on platform X with release Y? I would like to see the OKD configuration, exact version of ESX and vCenter and other parameters that impact the result. I would like to run your test myself.

BTW - @vrutkovs thanks so much for building OKD for us!

[vrutkovs]

Could you share the process to check the results of a specific build and test

Sure. Its a huge topic, but in short:

• every time a new OCP nightly is passing tests a new OKD nightly is being create at release controller
• example nightly status page
• in order to get accepted it needs to pass blocking job (aws) and few info jobs - upgrade from previous nightly and vsphere IPI install
• the test composes of few stages - usually its installation and conformance test. vSphere is failing most of the time as we use samples from Dockerhub and these may not be able to be imported (see these lines)

See job artifacts for a lot of data we collect during CI run:

• install-config.yaml used
• .openshift-install.log - this should contain most of the info I guess.

In any case, vSphere issues is something we want to have fixed, for best results try OKD 4.13 and file an issue with log-bundle - this should give us enough info to get started.

[pzi123]

@vrutkovs thanks for the release test details! Digesting that now. Just ran a fresh 4.13.0 build and opened #1673 with log bundle etc. Your statement "vSphere is failing most of the time" explains my findings. I guess this is the result of crossing boundaries between systems and cloud engineering. For a good measure I am compiling IPI failure data on 4.12 ... 4.9. We may catch the release when vsphere stopped working after 4.8. A related subject would be to start addressing UPI installs on vsphere. The documentation at https://docs.okd.io/4.13/installing/installing_vsphere/installing-vsphere.html#installation-infrastructure-user-infra_installing-vsphere shows signs of a drift with reality.

[pzi123]

Did most of the clean builds of 4.12 to 4.8 and I confirm that last working vsphere IPI is 4.8 - tried 4.8.0. and 4.8.57 both working but nothing above that release. Details at https://github.com/pzi123/tmp