[AWS] Cluster bootstrap fails with IngressStateEndpoints_MissingSubsets + operators not initialized

[dabcoder]

Similar to #1655 but we're on AWS.

Describe the bug
Cluster bootrstrap fails with the below errors:

ERROR Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthClientsController_SyncError::OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_SyncError::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::WellKnownReadyController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
ERROR OAuthClientsControllerDegraded: no ingress for host oauth-openshift.apps.openshift-self-hosted.sandbox....dev in route oauth-openshift in namespace openshift-authentication
ERROR OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to contain an admitted ingress: no admitted ingress for route oauth-openshift in namespace openshift-authentication
ERROR OAuthServerDeploymentDegraded:
ERROR OAuthServerRouteEndpointAccessibleControllerDegraded: route "openshift-authentication/oauth-openshift": status does not have a valid host address
ERROR OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.217.119:443/healthz": dial tcp 172.30.217.119:443: connect: connection refused
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
ERROR WellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this)
ERROR Cluster operator authentication Available is False with OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_ResourceNotFound::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes::WellKnown_NotReady: OAuthServerRouteEndpointAccessibleControllerAvailable: failed to retrieve route from cache: route.route.openshift.io "oauth-openshift" not found
ERROR OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.217.119:443/healthz": dial tcp 172.30.217.119:443: connect: connection refused
ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found
ERROR ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods).
ERROR WellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this)
INFO Cluster operator baremetal Disabled is False with :
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected
INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected
INFO Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected
INFO Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected
ERROR Cluster operator cluster-autoscaler Degraded is True with MissingDependency: machine-api not ready
ERROR Cluster operator console Degraded is True with ConsoleDefaultRouteSync_FailedAdmitDefaultRoute::DownloadsDefaultRouteSync_FailedAdmitDefaultRoute::RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedIngress: ConsoleDefaultRouteSyncDegraded: no ingress for host console-openshift-console.apps.openshift-self-hosted.sandbox.....dev in route console in namespace openshift-console
ERROR DownloadsDefaultRouteSyncDegraded: no ingress for host downloads-openshift-console.apps.openshift-self-hosted.sandbox.....dev in route downloads in namespace openshift-console
ERROR RouteHealthDegraded: console route is not admitted
ERROR SyncLoopRefreshDegraded: no ingress for host console-openshift-console.apps.openshift-self-hosted.sandbox.....dev in route console in namespace openshift-console
ERROR Cluster operator console Available is False with RouteHealth_RouteNotAdmitted: RouteHealthAvailable: console route is not admitted
ERROR Cluster operator control-plane-machine-set Available is False with UnavailableReplicas: Missing 3 available replica(s)
ERROR Cluster operator control-plane-machine-set Degraded is True with NoReadyMachines: No ready control plane machines found
INFO Cluster operator etcd RecentBackup is Unknown with ControllerStarted: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required
ERROR Cluster operator image-registry Available is False with DeploymentNotFound: Available: The deployment does not exist
ERROR NodeCADaemonAvailable: The daemon set node-ca has available replicas
ERROR ImagePrunerAvailable: Pruner CronJob has been created
INFO Cluster operator image-registry Progressing is True with Error: Progressing: Unable to apply resources: unable to sync storage configuration: unable to get cluster minted credentials "openshift-image-registry/installer-cloud-credentials": secret "installer-cloud-credentials" not found
INFO NodeCADaemonProgressing: The daemon set node-ca is deployed
ERROR Cluster operator image-registry Degraded is True with Unavailable: Degraded: The deployment does not exist
ERROR Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DNSReady=False (NoZones: The record isn't present in any zones.)
INFO Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
INFO ).
INFO Not all ingress controllers are available.
ERROR Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod "router-default-5cc984c688-76gsn" cannot be scheduled: 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Pod "router-default-5cc984c688-ghcgf" cannot be scheduled: 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Make sure you have sufficient worker nodes.), DNSReady=False (NoZones: The record isn't present in any zones.), CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)
INFO Cluster operator ingress EvaluationConditionsDetected is False with AsExpected:
INFO Cluster operator insights ClusterTransferAvailable is False with NoClusterTransfer: no available cluster transfer
INFO Cluster operator insights Disabled is False with AsExpected:
INFO Cluster operator insights SCAAvailable is Unknown with :
INFO Cluster operator machine-api Progressing is True with SyncingResources: Progressing towards operator: 4.13.0-0.okd-2023-08-18-135805
ERROR Cluster operator machine-api Degraded is True with SyncingFailed: Failed when progressing towards operator: 4.13.0-0.okd-2023-08-18-135805 because minimum worker replica count (2) not yet met: current running replicas 0, waiting for [openshift-self-hosted-hml2n-worker-us-west-2a-jmhsb openshift-self-hosted-hml2n-worker-us-west-2a-s5qtb]
ERROR Cluster operator machine-api Available is False with Initializing: Operator is initializing
ERROR Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: got 2 unavailable replicas
ERROR Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: got 2 unavailable replicas
INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.
INFO Cluster operator network ManagementStateDegraded is False with :
INFO Cluster operator network Progressing is True with Deploying: Deployment "/openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready
INFO Deployment "/openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
INFO Cluster operator storage Progressing is True with AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerServiceControllerProgressing: Waiting for Deployment to deploy pods
ERROR Cluster operator storage Available is False with AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying: AWSEBSCSIDriverOperatorCRAvailable: AWSEBSDriverControllerServiceControllerAvailable: Waiting for Deployment
ERROR Cluster initialization failed because one or more operators are not functioning properly.
ERROR The cluster should be accessible for troubleshooting as detailed in the documentation linked below,
ERROR https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html
ERROR The 'wait-for install-complete' subcommand can then be used to continue the installation
ERROR failed to initialize the cluster: Cluster operators authentication, console, control-plane-machine-set, image-registry, ingress, machine-api, monitoring, storage are not available

install-config.yaml:

apiVersion: v1
baseDomain: valid.domain.dev
credentialsMode: Manual
controlPlane:
  platform:
    aws:
      type: m6id.xlarge
      zones:
        - us-west-2a
  replicas: 3
compute: 
- name: worker
  platform:
    aws:
      type: m6id.xlarge
      zones:
        - us-west-2a
  replicas: 2
metadata:
  name: test-cluster-okd
platform:
  aws:
    region: us-west-2
    propagateUserTags: true
    userTags:
      adminContact: ...
pullSecret: '<valid-pull-secret>'
sshKey: "<valid-ssh-key>'

Version

openshift-install 4.13.0-0.okd-2023-08-18-135805
built from commit 26b887d89a13709170d9685ea1c133b3f865997d
release image quay.io/openshift/okd@sha256:59b1e486ed132fa084f654ca930d84a751c7956870f659ef522a9928bc200ec7
release architecture amd64

How reproducible

100% of the time.

Log bundle
The bootstrap machine is gone, so I can't generate those logs with ./openshift-install gather bootstrap --dir... if I am not mistaken? However, I can query logs with oc adm node-logs --role master....

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

ERROR Cluster operator machine-api Degraded is True with SyncingFailed: Failed when progressing towards operator: 4.13.0-0.okd-2023-08-18-135805 because minimum worker replica count (2) not yet met: current running replicas 0, waiting for [openshift-self-hosted-hml2n-worker-us-west-2a-jmhsb openshift-self-hosted-hml2n-worker-us-west-2a-s5qtb]

Seems two machines didn't come up? Have AWS created these machines?

[dabcoder]

Here are the logs @vrutkovs.

[dabcoder]

Seems two machines didn't come up? Have AWS created these machines?

No they didn't, and no EC2 instances were visible in the EC2 console. I'm trying to figure out why in AWS.

Please attach (or upload to the public file sharing service) must-gather archive

I've managed to produce those logs with the given command, thanks. Currently parsing those, will follow up here asap.

[vrutkovs]

Check Machine status in machine-api. Seems most failing operators are caused by workers missing - machine-api couldn't create them. control-plane-machine-set failing seems related to Machine API too

[dabcoder]

Just checking, were the provided logs any useful to pinpoint any issue?

[Felice-Copp56]

Hi, @dabcoder you have solved problem? i have the same issue

[dabcoder]

I'm afraid I did not @Felice-Copp56, I never got to the bottom of this issue.

[Felice-Copp56]

Hi @dabcoder thanks for your answer! I'm struggling with the issue and I hope that I can resolve it!

[sunruns]

same issue. how to resolve this?