Cluster Operators degarded after the upgrade from 4.10 to 4.11 #1564

imdmahajankanika · 2023-04-14T08:29:11Z

imdmahajankanika
Apr 14, 2023

Describe the bug

During the OKD upgrade from 4.10 --> 4.11, as soon as the MachineConfig operator started upgrading and started applying the upgrade on the master node, the console operator, authentication operator, openshift-apiserver operator etc. went down and even when all the nodes got upgraded and rebooted with the kubelet version (v1.24.6), the operators stayed downgraded et giving the errors like

APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503

APIServicesAvailable: "oauth.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request

NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.11.0-0.okd-2022-12-02-145640   False       False         True       52m     APIServicesAvailable: "oauth.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request...
baremetal                                  4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
cloud-controller-manager                   4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
cloud-credential                           4.11.0-0.okd-2022-12-02-145640   True        False         False      35d
cluster-autoscaler                         4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
config-operator                            4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
console                                    4.11.0-0.okd-2022-12-02-145640   False       False         True       51m     DeploymentAvailable: 0 replicas available for console deployment...
csi-snapshot-controller                    4.11.0-0.okd-2022-12-02-145640   True        False         False      4h35m
dns                                        4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
etcd                                       4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
image-registry                             4.11.0-0.okd-2022-12-02-145640   True        False         False      48m
ingress                                    4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
insights                                   4.11.0-0.okd-2022-12-02-145640   True        False         False      26s
kube-apiserver                             4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
kube-controller-manager                    4.11.0-0.okd-2022-12-02-145640   True        False         True       34d     GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp 172.30.124.81:9091: i/o timeout
kube-scheduler                             4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
kube-storage-version-migrator              4.11.0-0.okd-2022-12-02-145640   True        False         False      53m
machine-api                                4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
machine-approver                           4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
machine-config                             4.11.0-0.okd-2022-12-02-145640   True        False         False      24d
marketplace                                4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
monitoring                                 4.11.0-0.okd-2022-12-02-145640   False       True          True       39m     Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network                                    4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
node-tuning                                4.11.0-0.okd-2022-12-02-145640   True        False         False      76m
openshift-apiserver                        4.11.0-0.okd-2022-12-02-145640   False       False         False      42m     APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request...
openshift-controller-manager               4.11.0-0.okd-2022-12-02-145640   True        False         False      76m
openshift-samples                          4.11.0-0.okd-2022-12-02-145640   True        False         False      77m
operator-lifecycle-manager                 4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
operator-lifecycle-manager-catalog         4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
operator-lifecycle-manager-packageserver   4.11.0-0.okd-2022-12-02-145640   True        False         False      4m5s
service-ca                                 4.11.0-0.okd-2022-12-02-145640   True        False         False      34d
storage                                    4.11.0-0.okd-2022-12-02-145640   True        False         False      34d

Version

4.11.0-0.okd-2022-12-02-145640
Using Openshift SDN
UPI install method on Bare Metal servers (running on VMware Vsphere) hosted on OVH
ESXi 6.7 and later (VM version 14)

How reproducible

Upgrade the OKD cluster (Openshift SDN) from 4.10.0-0.okd-2022-07-09-073606 to 4.11.0-0.okd-2022-12-02-145640

Log bundle

davidson@services-test:~$ oc adm must-gather
[must-gather      ] OUT the server is currently unable to handle the request (get imagestreams.image.openshift.io must-gather)
[must-gather      ] OUT 
[must-gather      ] OUT Using must-gather plug-in image: registry.redhat.io/openshift4/ose-must-gather:latest
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: c4142fb9-f27b-46f0-861c-ac96577bec0f
ClusterVersion: Updating to "4.11.0-0.okd-2022-12-02-145640" from "4.10.0-0.okd-2022-07-09-073606" for 20 hours: Unable to apply 4.11.0-0.okd-2022-12-02-145640: wait has exceeded 40 minutes for these operators: kube-controller-manager
ClusterOperators:
	clusteroperator/authentication is not available (OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.166.159:443/healthz": dial tcp 172.30.166.159:443: i/o timeout (Client.Timeout exceeded while awaiting headers)) because OAuthClientsControllerDegraded: unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.166.159:443/healthz": dial tcp 172.30.166.159:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: [Get "https://10.130.0.4:6443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers), Get "https://10.129.0.27:6443/healthz": dial tcp 10.129.0.27:6443: i/o timeout (Client.Timeout exceeded while awaiting headers)]
	clusteroperator/console is not available (DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.outcastv2-test.davidson-idf.fr returns '503 Service Unavailable') because ConfigMapSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom)
DefaultRouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
OAuthClientSyncDegraded: oauth client for console does not exist and cannot be created (the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console))
RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.outcastv2-test.davidson-idf.fr returns '503 Service Unavailable'
	clusteroperator/image-registry is degraded because ImagePrunerDegraded: Job has reached the specified backoff limit
	clusteroperator/kube-controller-manager is degraded because GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
	clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating user workload thanos ruler: deleting Thanos Ruler Route failed: the server is currently unable to handle the request (delete routes.route.openshift.io thanos-ruler)
updating thanos querier: reconciling Thanos Querier Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io thanos-querier)
updating prometheus-k8s: reconciling Prometheus API Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io prometheus-k8s)
	clusteroperator/openshift-apiserver is not available (APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request) because All is well


[must-gather      ] OUT namespace/openshift-must-gather-jflfm created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-566q2 created
W0414 08:26:17.193900  307844 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "gather", "copy" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "gather", "copy" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "gather", "copy" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "gather", "copy" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[must-gather      ] OUT pod for plug-in image registry.redhat.io/openshift4/ose-must-gather:latest created
[must-gather-tslm6] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "registry.redhat.io/openshift4/ose-must-gather:latest"
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-566q2 deleted
[must-gather      ] OUT namespace/openshift-must-gather-jflfm deleted


When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: c4142fb9-f27b-46f0-861c-ac96577bec0f
ClusterVersion: Updating to "4.11.0-0.okd-2022-12-02-145640" from "4.10.0-0.okd-2022-07-09-073606" for 20 hours: Unable to apply 4.11.0-0.okd-2022-12-02-145640: wait has exceeded 40 minutes for these operators: kube-controller-manager
ClusterOperators:
	clusteroperator/authentication is not available (APIServicesAvailable: "oauth.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "user.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.166.159:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) because OAuthClientsControllerDegraded: unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.166.159:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: [Get "https://10.130.0.4:6443/healthz": dial tcp 10.130.0.4:6443: i/o timeout (Client.Timeout exceeded while awaiting headers), Get "https://10.129.0.27:6443/healthz": dial tcp 10.129.0.27:6443: i/o timeout (Client.Timeout exceeded while awaiting headers)]
	clusteroperator/console is not available (DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: console route is not admitted) because ConfigMapSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom)
DefaultRouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
OAuthClientSyncDegraded: oauth client for console does not exist and cannot be created (the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console))
RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
	clusteroperator/image-registry is degraded because ImagePrunerDegraded: Job has reached the specified backoff limit
	clusteroperator/kube-controller-manager is degraded because GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
	clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating user workload thanos ruler: deleting Thanos Ruler Route failed: the server is currently unable to handle the request (delete routes.route.openshift.io thanos-ruler)
updating thanos querier: reconciling Thanos Querier Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io thanos-querier)
updating prometheus-k8s: reconciling Prometheus API Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io prometheus-k8s)
	clusteroperator/openshift-apiserver is not available (APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request) because All is well

vrutkovs · 2023-04-14T08:31:19Z

vrutkovs
Apr 14, 2023
Maintainer

Please attach (or upload to the public file sharing service) must-gather archive, even if its incomplete

1 reply

imdmahajankanika Apr 14, 2023
Author

Below are the must-gather logs:-

davidson@services-test:~$ oc adm must-gather
[must-gather      ] OUT the server is currently unable to handle the request (get imagestreams.image.openshift.io must-gather)
[must-gather      ] OUT 
[must-gather      ] OUT Using must-gather plug-in image: registry.redhat.io/openshift4/ose-must-gather:latest
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: c4142fb9-f27b-46f0-861c-ac96577bec0f
ClusterVersion: Updating to "4.11.0-0.okd-2022-12-02-145640" from "4.10.0-0.okd-2022-07-09-073606" for 20 hours: Unable to apply 4.11.0-0.okd-2022-12-02-145640: wait has exceeded 40 minutes for these operators: kube-controller-manager
ClusterOperators:
	clusteroperator/authentication is not available (OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.166.159:443/healthz": dial tcp 172.30.166.159:443: i/o timeout (Client.Timeout exceeded while awaiting headers)) because OAuthClientsControllerDegraded: unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.166.159:443/healthz": dial tcp 172.30.166.159:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: [Get "https://10.130.0.4:6443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers), Get "https://10.129.0.27:6443/healthz": dial tcp 10.129.0.27:6443: i/o timeout (Client.Timeout exceeded while awaiting headers)]
	clusteroperator/console is not available (DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.outcastv2-test.davidson-idf.fr returns '503 Service Unavailable') because ConfigMapSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom)
DefaultRouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
OAuthClientSyncDegraded: oauth client for console does not exist and cannot be created (the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console))
RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.outcastv2-test.davidson-idf.fr returns '503 Service Unavailable'
	clusteroperator/image-registry is degraded because ImagePrunerDegraded: Job has reached the specified backoff limit
	clusteroperator/kube-controller-manager is degraded because GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
	clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating user workload thanos ruler: deleting Thanos Ruler Route failed: the server is currently unable to handle the request (delete routes.route.openshift.io thanos-ruler)
updating thanos querier: reconciling Thanos Querier Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io thanos-querier)
updating prometheus-k8s: reconciling Prometheus API Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io prometheus-k8s)
	clusteroperator/openshift-apiserver is not available (APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request) because All is well


[must-gather      ] OUT namespace/openshift-must-gather-jflfm created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-566q2 created
W0414 08:26:17.193900  307844 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "gather", "copy" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "gather", "copy" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "gather", "copy" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "gather", "copy" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[must-gather      ] OUT pod for plug-in image registry.redhat.io/openshift4/ose-must-gather:latest created
[must-gather-tslm6] OUT gather did not start: unable to pull image: ImagePullBackOff: Back-off pulling image "registry.redhat.io/openshift4/ose-must-gather:latest"
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-566q2 deleted
[must-gather      ] OUT namespace/openshift-must-gather-jflfm deleted


When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information.
ClusterID: c4142fb9-f27b-46f0-861c-ac96577bec0f
ClusterVersion: Updating to "4.11.0-0.okd-2022-12-02-145640" from "4.10.0-0.okd-2022-07-09-073606" for 20 hours: Unable to apply 4.11.0-0.okd-2022-12-02-145640: wait has exceeded 40 minutes for these operators: kube-controller-manager
ClusterOperators:
	clusteroperator/authentication is not available (APIServicesAvailable: "oauth.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "user.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.166.159:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) because OAuthClientsControllerDegraded: unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (post oauthclients.oauth.openshift.io)
OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.outcastv2-test.davidson-idf.fr/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.166.159:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: [Get "https://10.130.0.4:6443/healthz": dial tcp 10.130.0.4:6443: i/o timeout (Client.Timeout exceeded while awaiting headers), Get "https://10.129.0.27:6443/healthz": dial tcp 10.129.0.27:6443: i/o timeout (Client.Timeout exceeded while awaiting headers)]
	clusteroperator/console is not available (DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: console route is not admitted) because ConfigMapSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)
CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom)
DefaultRouteSyncDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
OAuthClientSyncDegraded: oauth client for console does not exist and cannot be created (the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console))
RouteHealthDegraded: console route is not admitted
SyncLoopRefreshDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
	clusteroperator/image-registry is degraded because ImagePrunerDegraded: Job has reached the specified backoff limit
	clusteroperator/kube-controller-manager is degraded because GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
	clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating user workload thanos ruler: deleting Thanos Ruler Route failed: the server is currently unable to handle the request (delete routes.route.openshift.io thanos-ruler)
updating thanos querier: reconciling Thanos Querier Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io thanos-querier)
updating prometheus-k8s: reconciling Prometheus API Route failed: retrieving Route object failed: the server is currently unable to handle the request (get routes.route.openshift.io prometheus-k8s)
	clusteroperator/openshift-apiserver is not available (APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request) because All is well


Gathering data for ns/openshift-config...
Gathering data for ns/openshift-config-managed...
Gathering data for ns/openshift-authentication...
Gathering data for ns/openshift-authentication-operator...
Gathering data for ns/openshift-ingress...
I0414 08:40:43.245896  307844 request.go:665] Waited for 1.107869789s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/apps.kio.kasten.io/v1alpha1?timeout=32s
I0414 08:40:54.046172  307844 request.go:665] Waited for 1.191061346s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/security.openshift.io/v1?timeout=32s
Gathering data for ns/openshift-oauth-apiserver...
I0414 08:41:11.227552  307844 request.go:665] Waited for 1.108934564s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/operator.openshift.io/v1?timeout=32s
I0414 08:41:22.028190  307844 request.go:665] Waited for 1.187014008s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/user.openshift.io/v1?timeout=32s
Gathering data for ns/openshift-machine-api...
I0414 08:41:38.758634  307844 request.go:665] Waited for 1.115668032s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I0414 08:41:49.958744  307844 request.go:665] Waited for 1.190774847s due to client-side throttling, not priority and fairness, request: GET:https://api.outcastv2-test.davidson-idf.fr:6443/apis/packages.operators.coreos.com/v1?timeout=32s

vrutkovs · 2023-04-14T08:31:41Z

vrutkovs
Apr 14, 2023
Maintainer

Possibly a dupe of #1550?

2 replies

imdmahajankanika Apr 14, 2023
Author

@vrutkovs #1550 is for version 4.11.0-0.okd-2023-01-14-152430 and using OVN. This issue is about 4.11.0-0.okd-2022-12-02-145640 (Openshift SDN). And, as the cluster operators are degraded, we are unable to access web console, test application deployments and pod-to-pod communication.

This might be an related issue but not a duplicate of #1550

yuanlinios Jul 14, 2023

I met the same issue, but found it was just a dupe of #1550. After I applied the workaround to fix selinux, the cluster recovered.

You can test pod-to-pod communication from CLI first

jdhirst · 2023-04-20T10:44:16Z

jdhirst
Apr 20, 2023

@vrutkovs I have also encountered the same problem here during an OCP upgrade from 4.12.12 to 4.12.13:

CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom)
must-gather: https://mega.nz/file/fLpWDZQI#pWwAc3eTpo7AcpsvcNmJtyYVUcxOjb4nPatCGQ64eyE

This is a SNO cluster with OVNKubernetes instead of OpenShiftSDN.

2 replies

yocum137 May 6, 2023

Same, upgrading SNO 4.12.10 -> 4.12.14

dmoessne May 9, 2023

run into the same from 4.11.38 -> 4.12.14. (SNO OVNKubernetes)
https://access.redhat.com/solutions/6999944 solved the issue for me

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster Operators degarded after the upgrade from 4.10 to 4.11 #1564

{{title}}

Replies: 3 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cluster Operators degarded after the upgrade from 4.10 to 4.11 #1564

imdmahajankanika Apr 14, 2023

Replies: 3 comments · 5 replies

vrutkovs Apr 14, 2023 Maintainer

imdmahajankanika Apr 14, 2023 Author

vrutkovs Apr 14, 2023 Maintainer

imdmahajankanika Apr 14, 2023 Author

yuanlinios Jul 14, 2023

jdhirst Apr 20, 2023

yocum137 May 6, 2023

dmoessne May 9, 2023

imdmahajankanika
Apr 14, 2023

Replies: 3 comments 5 replies

vrutkovs
Apr 14, 2023
Maintainer

imdmahajankanika Apr 14, 2023
Author

vrutkovs
Apr 14, 2023
Maintainer

imdmahajankanika Apr 14, 2023
Author

jdhirst
Apr 20, 2023