OKD 4.13 can not access to web console #1742

balioune · 2023-09-19T09:36:30Z

balioune
Sep 19, 2023

Hi,
I installed an OKD cluster lab with one master and two nodes, I configure DNS wilcard to point to the master.
All services are availables except the Openshift web console
The command "oc describe pod console-65d475d657-bnjn8 -n openshift-console" shows that livenessProbe and readinessProbe try to chech healthy port 8443 which seems

In my understanding the console pods don't start because their probes are always failing.

I think the main issue is the access of this route oauth-openshift.apps.ocp4-kubevirt01.fr.intranet and the canary
Any suggestion to solve this issue ?
BR,

$ ./oc get all -n openshift-authentication
NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
pod/oauth-openshift-68db65946f-z74n8   1/1     Running   0          17h   10.128.0.91   host-10-253-235-58   <none>           <none>

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/oauth-openshift   ClusterIP   172.30.149.253   <none>        443/TCP   20h   app=oauth-openshift

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS        IMAGES                                                                                                  SELECTOR
deployment.apps/oauth-openshift   1/1     1            1           17h   oauth-openshift   quay.io/openshift/okd-content@sha256:a7260990cd366b7cd382aa1acef71cf9f0d6afc0f22b7d36ddc9bb593ab76784   app=oauth-openshift

NAME                                         DESIRED   CURRENT   READY   AGE   CONTAINERS        IMAGES                                                                                                  SELECTOR
replicaset.apps/oauth-openshift-56dccc9644   0         0         0       17h   oauth-openshift   quay.io/openshift/okd-content@sha256:a7260990cd366b7cd382aa1acef71cf9f0d6afc0f22b7d36ddc9bb593ab76784   app=oauth-openshift,pod-template-hash=56dccc9644
replicaset.apps/oauth-openshift-68db65946f   1         1         1       17h   oauth-openshift   quay.io/openshift/okd-content@sha256:a7260990cd366b7cd382aa1acef71cf9f0d6afc0f22b7d36ddc9bb593ab76784   app=oauth-openshift,pod-template-hash=68db65946f

NAME                                       HOST/PORT                                          PATH   SERVICES          PORT   TERMINATION            WILDCARD
route.route.openshift.io/oauth-openshift   oauth-openshift.apps.ocp4-kubevirt01.fr.intranet          oauth-openshift   6443   passthrough/Redirect   None


$ ./oc describe route oauth-openshift -n openshift-authentication
Name:                   oauth-openshift
Namespace:              openshift-authentication
Created:                20 hours ago
Labels:                 app=oauth-openshift
Annotations:            <none>
Requested Host:         oauth-openshift.apps.ocp4-kubevirt01.fr.intranet
                           exposed on router default (host router-default.apps.ocp4-kubevirt01.fr.intranet) 18 hours ago
Path:                   <none>
TLS Termination:        passthrough
Insecure Policy:        Redirect
Endpoint Port:          6443

Service:        oauth-openshift
Weight:         100 (100%)
Endpoints:      10.128.0.91:6443

$ oc get clusteroperator
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.0-0.okd-2023-09-03-082426   False       False         True       20h     OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ocp4-kubevirt01.fr.intranet/healthz": dial tcp 10.253.235.58:443: connect: connection refused
baremetal                                  4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
cloud-controller-manager                   4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
cloud-credential                           4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
cluster-autoscaler                         4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
config-operator                            4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
console                                    4.13.0-0.okd-2023-09-03-082426   False       True          False      19h     DeploymentAvailable: 0 replicas available for console deployment...
control-plane-machine-set                  4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
csi-snapshot-controller                    4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
dns                                        4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
etcd                                       4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
image-registry                             4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
ingress                                    4.13.0-0.okd-2023-09-03-082426   True        False         True       19h     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
insights                                   4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
kube-apiserver                             4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
kube-controller-manager                    4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
kube-scheduler                             4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
kube-storage-version-migrator              4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
machine-api                                4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
machine-approver                           4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
machine-config                             4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
marketplace                                4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
monitoring                                 4.13.0-0.okd-2023-09-03-082426   True        False         False      17h
network                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
node-tuning                                4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
openshift-apiserver                        4.13.0-0.okd-2023-09-03-082426   True        False         False      81m
openshift-controller-manager               4.13.0-0.okd-2023-09-03-082426   True        False         False      82m
openshift-samples                          4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
operator-lifecycle-manager                 4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
operator-lifecycle-manager-catalog         4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
operator-lifecycle-manager-packageserver   4.13.0-0.okd-2023-09-03-082426   True        False         False      19h
service-ca                                 4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
storage                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      20h

$ ./oc adm must-gather
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift/okd-content@sha256:2de098371b3a9237868f53a1549273ab33f72b4f5a126b26aa4dfe45903a59e5
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 4b925f52-aa1f-412d-bcc1-d61772e79f13
ClusterVersion: Installing "4.13.0-0.okd-2023-09-03-082426" for 21 hours: Unable to apply 4.13.0-0.okd-2023-09-03-082426: some cluster operators are not available
ClusterOperators:
        clusteroperator/authentication is not available (OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ocp4-kubevirt01.fr.intranet/healthz": dial tcp 10.253.235.58:443: connect: connection refused) because OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.ocp4-kubevirt01.fr.intranet/healthz": dial tcp 10.253.235.58:443: connect: connection refused
        clusteroperator/console is not available (DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.ocp4-kubevirt01.fr.intranet): Get "https://console-openshift-console.apps.ocp4-kubevirt01.fr.intranet": dial tcp 10.253.235.58:443: connect: connection refused) because RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.ocp4-kubevirt01.fr.intranet): Get "https://console-openshift-console.apps.ocp4-kubevirt01.fr.intranet": dial tcp 10.253.235.58:443: connect: connection refused
        clusteroperator/ingress is degraded because The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

Answered by vrutkovs

Sep 19, 2023

Seems the cluster never completed the installation, so you'll need to link to a log bundle

auth and console never rollout because ingress is not ready, which throws

CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

That means "ingress can't reach to the pods", you may want to check your never connectivity between nodes

View full answer

vrutkovs · 2023-09-19T11:02:49Z

vrutkovs
Sep 19, 2023
Maintainer

Seems the cluster never completed the installation, so you'll need to link to a log bundle

auth and console never rollout because ingress is not ready, which throws

CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

That means "ingress can't reach to the pods", you may want to check your never connectivity between nodes

0 replies

balioune · 2023-09-19T12:45:10Z

balioune
Sep 19, 2023
Author

@vrutkovs thanks for you reply.
I solve the issue, the ingress-controller running on the master

 oc get clusteroperator
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.0-0.okd-2023-09-03-082426   True        False         False      57m
baremetal                                  4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
cloud-controller-manager                   4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
cloud-credential                           4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
cluster-autoscaler                         4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
config-operator                            4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
console                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      80m
control-plane-machine-set                  4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
csi-snapshot-controller                    4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
dns                                        4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
etcd                                       4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
image-registry                             4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
ingress                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
insights                                   4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
kube-apiserver                             4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
kube-controller-manager                    4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
kube-scheduler                             4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
kube-storage-version-migrator              4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
machine-api                                4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
machine-approver                           4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
machine-config                             4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
marketplace                                4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
monitoring                                 4.13.0-0.okd-2023-09-03-082426   True        False         False      20h
network                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
node-tuning                                4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
openshift-apiserver                        4.13.0-0.okd-2023-09-03-082426   True        False         False      59m
openshift-controller-manager               4.13.0-0.okd-2023-09-03-082426   True        False         False      4h21m
openshift-samples                          4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
operator-lifecycle-manager                 4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
operator-lifecycle-manager-catalog         4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
operator-lifecycle-manager-packageserver   4.13.0-0.okd-2023-09-03-082426   True        False         False      22h
service-ca                                 4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
storage                                    4.13.0-0.okd-2023-09-03-082426   True        False         False      23h
[openstack@openstackain1 okd_install]$ oc get clusterversion
NAME      VERSION                          AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-0.okd-2023-09-03-082426   True        False         80m     Cluster version is 4.13.0-0.okd-2023-09-03-082426

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OKD 4.13 can not access to web console #1742

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

OKD 4.13 can not access to web console #1742

balioune Sep 19, 2023

Replies: 2 comments

vrutkovs Sep 19, 2023 Maintainer

balioune Sep 19, 2023 Author

balioune
Sep 19, 2023

vrutkovs
Sep 19, 2023
Maintainer

balioune
Sep 19, 2023
Author