OKD install fail #813

xiaosayishi · 2021-08-16T10:48:35Z

xiaosayishi
Aug 16, 2021

When I installed OKD, boostrap has started normally. When I installed 3 masters, the following questions were always prompted：

[root@bastion ~]# openshift-install wait-for install-complete --log-level=debug --dir=/okdinstall
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.okd.test1.com:6443 to initialize...
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:

Cluster operator etcd is reporting a failure: NodeInstallerDegraded: 1 nodes are failing on revision 2:
NodeInstallerDegraded:
StaticPodsDegraded: pods "etcd-master1.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master2.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master3.okd.test1.com" not found
Cluster operator kube-apiserver is reporting a failure: StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is not ready: unknown reason
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
NodeInstallerDegraded: 1 nodes are failing on revision 27:
NodeInstallerDegraded: ; 1 nodes are failing on revision 28:
NodeInstallerDegraded: static pod of revision 28 has been installed, but is not ready while new revision 29 is pending; 1 nodes are failing on revision 29:
NodeInstallerDegraded: static pod of revision 29 has been installed, but is not ready while new revision 30 is pending
Could not update oauthclient "console" (363 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (198 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (315 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-authentication-operator/authentication-operator" (510 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-machine-approver/cluster-machine-approver" (520 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-version/cluster-version-operator" (8 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-config-operator/config-operator" (124 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-image-registry/image-registry" (516 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (230 of 596): the server does not recognize this resource, check extension API servers
E0816 18:05:59.445622 27751 reflector.go:307] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=179203&timeoutSeconds=585&watch=true: net/http: TLS handshake timeout
E0816 18:06:04.329983 27751 reflector.go:153] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&limit=500&resourceVersion=0: EOF
E0816 18:06:08.336685 27751 reflector.go:153] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&limit=500&resourceVersion=0: EOF
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Cluster operator etcd is reporting a failure: NodeInstallerDegraded: 1 nodes are failing on revision 2:
NodeInstallerDegraded:
StaticPodsDegraded: pods "etcd-master1.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master2.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master3.okd.test1.com" not found
Cluster operator kube-apiserver is reporting a failure: StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is not ready: unknown reason
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
NodeInstallerDegraded: 1 nodes are failing on revision 27:
NodeInstallerDegraded: ; 1 nodes are failing on revision 28:
NodeInstallerDegraded: static pod of revision 28 has been installed, but is not ready while new revision 29 is pending; 1 nodes are failing on revision 29:
NodeInstallerDegraded: static pod of revision 29 has been installed, but is not ready while new revision 30 is pending
Could not update oauthclient "console" (363 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (198 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (315 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-authentication-operator/authentication-operator" (510 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-machine-approver/cluster-machine-approver" (520 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-version/cluster-version-operator" (8 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-config-operator/config-operator" (124 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-image-registry/image-registry" (516 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (230 of 596): the server does not recognize this resource, check extension API servers
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Cluster operator etcd is reporting a failure: NodeInstallerDegraded: 1 nodes are failing on revision 2:
NodeInstallerDegraded:
StaticPodsDegraded: pods "etcd-master1.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master2.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master3.okd.test1.com" not found
Cluster operator kube-apiserver is reporting a failure: StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master2.okd.test1.com_openshift-kube-apiserver(e5a30b976c93d83eb6978761478ec71e)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master2.okd.test1.com_openshift-kube-apiserver(e5a30b976c93d83eb6978761478ec71e)
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is not ready: unknown reason
NodeInstallerDegraded: 1 nodes are failing on revision 27:
NodeInstallerDegraded: ; 1 nodes are failing on revision 28:
NodeInstallerDegraded: static pod of revision 28 has been installed, but is not ready while new revision 29 is pending; 1 nodes are failing on revision 29:
NodeInstallerDegraded: static pod of revision 29 has been installed, but is not ready while new revision 30 is pending
Could not update oauthclient "console" (363 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (198 of 596): the server does not recognize this resource, check extension API servers
Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (315 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-authentication-operator/authentication-operator" (510 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-machine-approver/cluster-machine-approver" (520 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-cluster-version/cluster-version-operator" (8 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-config-operator/config-operator" (124 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-image-registry/image-registry" (516 of 596): the server does not recognize this resource, check extension API servers
Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (230 of 596): the server does not recognize this resource, check extension API servers
E0816 18:11:25.382321 27751 reflector.go:307] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dversion&resourceVersion=180894&timeoutSeconds=318&watch=true: net/http: TLS handshake timeout
E0816 18:11:29.243329 27751 reflector.go:153] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&limit=500&resourceVersion=0: EOF
E0816 18:11:33.248925 27751 reflector.go:153] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to list *v1.ClusterVersion: Get https://api.okd.test1.com:6443/apis/config.openshift.io/v1/clusterversions?fieldSelector=metadata.name%3Dversion&limit=500&resourceVersion=0: EOF
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 11% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 54% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 61% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 67% complete
E0816 18:12:27.131548 27751 reflector.go:307] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: the server is currently unable to handle the request (get clusterversions.config.openshift.io)
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 67% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 68% complete
DEBUG Still waiting for the cluster to initialize: Multiple errors are preventing progress:
Cluster operator etcd is reporting a failure: NodeInstallerDegraded: 1 nodes are failing on revision 2:
NodeInstallerDegraded:
StaticPodsDegraded: pods "etcd-master1.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master2.okd.test1.com" not found
StaticPodsDegraded: pods "etcd-master3.okd.test1.com" not found
Cluster operator kube-apiserver is reporting a failure: StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master1.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1.okd.test1.com_openshift-kube-apiserver(279d539d2b829e48954c5e1ea7715e07)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master2.okd.test1.com_openshift-kube-apiserver(e5a30b976c93d83eb6978761478ec71e)
StaticPodsDegraded: pod/kube-apiserver-master2.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master2.okd.test1.com_openshift-kube-apiserver(e5a30b976c93d83eb6978761478ec71e)
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is not ready: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
StaticPodsDegraded: pod/kube-apiserver-master3.okd.test1.com container "kube-apiserver" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master3.okd.test1.com_openshift-kube-apiserver(8b69b74bf6c2252a805bd0e28823cbf8)
NodeInstallerDegraded: 1 nodes are failing on revision 27:
NodeInstallerDegraded: ; 1 nodes are failing on revision 28:
NodeInstallerDegraded: static pod of revision 28 has been installed, but is not ready while new revision 29 is pending; 1 nodes are failing on revision 29:

Status of the three master nodes：

log-bundle-20210205220425.zip

[root@bastion ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
master1.okd.test1.com Ready master 7h17m v1.18.3
master2.okd.test1.com Ready master 6h28m v1.18.3
master3.okd.test1.com Ready master 5h43m v1.18.3

[root@bastion ~]# oc get NAMESPACE openshift-apiserver-operator openshift-authentication-operator openshift-cluster-machine-approver openshift-cluster-node-tuning-operator openshift-cluster-node-tuning-operator openshift-cluster-node-tuning-operator openshift-cluster-node-tuning-operator openshift-cluster-storage-operator openshift-cluster-version openshift-controller-manager-operator openshift-dns-operator openshift-dns openshift-dns openshift-dns openshift-etcd-operator openshift-etcd openshift-etcd openshift-etcd openshift-kube-apiserver-operator openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-apiserver openshift-kube-controller-manager-operator openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-controller-manager openshift-kube-scheduler-operator openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-scheduler openshift-kube-storage-version-mi openshift-kube-storage-version-migrator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-machine-config-operator openshift-multus openshift-multus openshift-multus openshift-multus openshift-multus openshift-multus openshift-network-operator openshift-operator-lifecycle-manager openshift-operator-lifecycle-manager openshift-operator-lifecycle-manager openshift-operator-lifecycle-manager openshift-operator-lifecycle-manager openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-sdn openshift-service-ca-operator openshift-service-ca pods --all-namespaces
NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator-58d685cd8f-pdwqt 1/1 Running 1 6h30m
authentication-operator-7dcb56f456-5dtwz 1/1 Running 31 6h29m
machine-approver-86756d9f66-bcw48 2/2 Running 0 6h29m
cluster-node-tuning-operator-5f6978996c-4jwlm 1/1 Running 0 6h29m
tuned-2spm4 1/1 Running 0 5h39m
tuned-s5p4b 1/1 Running 0 5h2m
tuned-x4zht 1/1 Running 0 4h17m
csi-snapshot-controller-operator-58b6b7dc6b-w5cr6 1/1 Running 0 6h29m
cluster-version-operator-8b7674d7d-wgcnk 1/1 Running 0 6h30m
openshift-controller-manager-operator-6ff4fc8498-rhc4m 1/1 Running 1 6h29m
dns-operator-74b5fcd76f-d9gm5 2/2 Running 0 6h30m
dns-default-kg5jz 3/3 Running 0 4h17m
dns-default-t9298 3/3 Running 0 5h2m
dns-default-zfzpr 3/3 Running 0 5h36m
etcd-operator-78bc85bcf5-hnthg 1/1 Running 1 6h29m
installer-2-master1.okd.test1.com 0/1 Completed 0 4h17m
installer-3-master1.okd.test1.com 0/1 Completed 0 58m
revision-pruner-2-master1.okd.test1.com 0/1 Completed 0 58m
kube-apiserver-operator-f4bd94d79-9wnqp 1/1 Running 1 6h29m
installer-22-master1.okd.test1.com 0/1 Completed 0 65m
installer-23-master2.okd.test1.com 0/1 Completed 0 46m
installer-24-master3.okd.test1.com 0/1 Completed 0 26m
installer-25-master1.okd.test1.com 0/1 Completed 0 24m
installer-26-master2.okd.test1.com 0/1 Completed 0 5m58s
installer-27-master3.okd.test1.com 0/1 Completed 0 4m31s
kube-apiserver-master1.okd.test1.com 3/4 CrashLoopBackOff 9 24m
kube-apiserver-master2.okd.test1.com 3/4 CrashLoopBackOff 5 5m35s
kube-apiserver-master3.okd.test1.com 3/4 CrashLoopBackOff 4 4m21s
revision-pruner-22-master1.okd.test1.com 0/1 Completed 0 46m
revision-pruner-23-master2.okd.test1.com 0/1 Completed 0 26m
revision-pruner-24-master3.okd.test1.com 0/1 Completed 0 25m
revision-pruner-25-master1.okd.test1.com 0/1 Completed 0 6m13s
revision-pruner-26-master2.okd.test1.com 0/1 Completed 0 4m44s
kube-controller-manager-operator-5cc64658f8-dlkwd 1/1 Running 1 6h30m
installer-21-master1.okd.test1.com 0/1 Completed 0 87m
installer-21-master2.okd.test1.com 0/1 Completed 0 86m
installer-22-master1.okd.test1.com 0/1 Completed 0 84m
installer-22-master2.okd.test1.com 0/1 Completed 0 86m
installer-22-master3.okd.test1.com 0/1 Completed 0 85m
installer-23-master1.okd.test1.com 0/1 Completed 0 67m
installer-23-master2.okd.test1.com 0/1 Completed 0 66m
installer-24-master1.okd.test1.com 0/1 Completed 0 65m
installer-24-master2.okd.test1.com 0/1 Completed 0 66m
installer-24-master3.okd.test1.com 0/1 Completed 0 65m
installer-25-master1.okd.test1.com 0/1 Completed 0 47m
installer-25-master2.okd.test1.com 0/1 Completed 0 46m
installer-26-master1.okd.test1.com 0/1 Completed 0 44m
installer-26-master2.okd.test1.com 0/1 Completed 0 46m
installer-26-master3.okd.test1.com 0/1 Completed 0 45m
installer-27-master1.okd.test1.com 0/1 Completed 0 26m
installer-27-master2.okd.test1.com 0/1 Completed 0 26m
installer-28-master1.okd.test1.com 0/1 Completed 0 23m
installer-28-master2.okd.test1.com 0/1 Completed 0 25m
installer-28-master3.okd.test1.com 0/1 Completed 0 24m
installer-29-master1.okd.test1.com 0/1 Completed 0 5m37s
installer-29-master2.okd.test1.com 0/1 Completed 0 6m31s
installer-30-master1.okd.test1.com 0/1 Completed 0 5m13s
installer-30-master2.okd.test1.com 0/1 Completed 0 3m47s
installer-30-master3.okd.test1.com 0/1 Completed 0 4m31s
kube-controller-manager-master1.okd.test1.com 4/4 Running 1 5m7s
kube-controller-manager-master2.okd.test1.com 4/4 Running 0 3m40s
kube-controller-manager-master3.okd.test1.com 4/4 Running 0 4m24s
revision-pruner-21-master1.okd.test1.com 0/1 Completed 0 86m
revision-pruner-21-master2.okd.test1.com 0/1 Completed 0 86m
revision-pruner-22-master1.okd.test1.com 0/1 Completed 0 84m
revision-pruner-22-master2.okd.test1.com 0/1 Completed 0 85m
revision-pruner-22-master3.okd.test1.com 0/1 Completed 0 84m
revision-pruner-23-master1.okd.test1.com 0/1 Completed 0 67m
revision-pruner-23-master2.okd.test1.com 0/1 Completed 0 66m
revision-pruner-24-master1.okd.test1.com 0/1 Completed 0 64m
revision-pruner-24-master2.okd.test1.com 0/1 Completed 0 65m
revision-pruner-24-master3.okd.test1.com 0/1 Completed 0 65m
revision-pruner-25-master1.okd.test1.com 0/1 Completed 0 46m
revision-pruner-25-master2.okd.test1.com 0/1 Completed 0 46m
revision-pruner-26-master1.okd.test1.com 0/1 Completed 0 44m
revision-pruner-26-master2.okd.test1.com 0/1 Completed 0 45m
revision-pruner-26-master3.okd.test1.com 0/1 Completed 0 44m
revision-pruner-27-master1.okd.test1.com 0/1 Completed 0 26m
revision-pruner-27-master2.okd.test1.com 0/1 Completed 0 25m
revision-pruner-28-master1.okd.test1.com 0/1 Completed 0 23m
revision-pruner-28-master2.okd.test1.com 0/1 Completed 0 24m
revision-pruner-28-master3.okd.test1.com 0/1 Completed 0 24m
revision-pruner-29-master1.okd.test1.com 0/1 Completed 0 5m15s
revision-pruner-29-master2.okd.test1.com 0/1 Completed 0 5m37s
revision-pruner-30-master1.okd.test1.com 0/1 Completed 0 4m43s
revision-pruner-30-master2.okd.test1.com 0/1 Completed 0 3m16s
revision-pruner-30-master3.okd.test1.com 0/1 Completed 0 3m57s
openshift-kube-scheduler-operator-7b68c8c947-w94zm 1/1 Running 1 6h30m
installer-19-master1.okd.test1.com 0/1 Completed 0 108m
installer-19-master2.okd.test1.com 0/1 Completed 0 107m
installer-21-master1.okd.test1.com 0/1 Completed 0 87m
installer-21-master2.okd.test1.com 0/1 Completed 0 86m
installer-22-master1.okd.test1.com 0/1 Completed 0 83m
installer-22-master2.okd.test1.com 0/1 Completed 0 86m
installer-22-master3.okd.test1.com 0/1 Completed 0 85m
installer-23-master1.okd.test1.com 0/1 Completed 0 67m
installer-24-master1.okd.test1.com 0/1 Completed 0 66m
installer-24-master2.okd.test1.com 0/1 Completed 0 65m
installer-24-master3.okd.test1.com 0/1 Completed 0 63m
installer-25-master1.okd.test1.com 0/1 Completed 0 47m
installer-26-master1.okd.test1.com 0/1 Completed 0 46m
installer-26-master2.okd.test1.com 0/1 Completed 0 45m
installer-26-master3.okd.test1.com 0/1 Completed 0 43m
installer-27-master1.okd.test1.com 0/1 Completed 0 26m
installer-27-master2.okd.test1.com 0/1 Completed 0 25m
installer-28-master1.okd.test1.com 0/1 Completed 0 22m
installer-28-master2.okd.test1.com 0/1 Completed 0 25m
installer-28-master3.okd.test1.com 0/1 Completed 0 24m
installer-29-master1.okd.test1.com 0/1 Completed 0 6m37s
installer-30-master1.okd.test1.com 0/1 Completed 0 3m
installer-30-master2.okd.test1.com 0/1 Completed 0 5m22s
installer-30-master3.okd.test1.com 0/1 Completed 0 4m10s
openshift-kube-scheduler-master1.okd.test1.com 2/2 Running 0 2m55s
openshift-kube-scheduler-master2.okd.test1.com 2/2 Running 0 5m7s
openshift-kube-scheduler-master3.okd.test1.com 2/2 Running 0 4m5s
revision-pruner-19-master1.okd.test1.com 0/1 Completed 0 107m
revision-pruner-19-master2.okd.test1.com 0/1 Completed 0 106m
revision-pruner-21-master1.okd.test1.com 0/1 Completed 0 86m
revision-pruner-21-master2.okd.test1.com 0/1 Completed 0 86m
revision-pruner-22-master1.okd.test1.com 0/1 Completed 0 83m
revision-pruner-22-master2.okd.test1.com 0/1 Completed 0 85m
revision-pruner-22-master3.okd.test1.com 0/1 Completed 0 84m
revision-pruner-23-master1.okd.test1.com 0/1 Completed 0 66m
revision-pruner-24-master1.okd.test1.com 0/1 Completed 0 65m
revision-pruner-24-master2.okd.test1.com 0/1 Completed 0 63m
revision-pruner-24-master3.okd.test1.com 0/1 Completed 0 62m
revision-pruner-25-master1.okd.test1.com 0/1 Completed 0 46m
revision-pruner-26-master1.okd.test1.com 0/1 Completed 0 45m
revision-pruner-26-master2.okd.test1.com 0/1 Completed 0 44m
revision-pruner-26-master3.okd.test1.com 0/1 Completed 0 42m
revision-pruner-27-master1.okd.test1.com 0/1 Completed 0 25m
revision-pruner-27-master2.okd.test1.com 0/1 Completed 0 25m
revision-pruner-28-master1.okd.test1.com 0/1 Completed 0 22m
revision-pruner-28-master2.okd.test1.com 0/1 Completed 0 24m
revision-pruner-28-master3.okd.test1.com 0/1 Completed 0 23m
revision-pruner-29-master1.okd.test1.com 0/1 Completed 0 5m33s
revision-pruner-30-master1.okd.test1.com 0/1 Completed 0 119s
revision-pruner-30-master2.okd.test1.com 0/1 Completed 0 4m15s
revision-pruner-30-master3.okd.test1.com 0/1 Completed 0 3m9s
grator-operator kube-storage-version-migrator-operator-644d874cd8-s6d76 1/1 Running 2 6h29m
migrator-6879df9b64-k4tv9 0/1 Pending 0 5h40m
etcd-quorum-guard-7bb76959df-69mpv 0/1 Running 0 5h42m
etcd-quorum-guard-7bb76959df-bghgq 0/1 Running 0 5h42m
etcd-quorum-guard-7bb76959df-fjrbz 0/1 Preempting 0 5h42m
etcd-quorum-guard-7bb76959df-q2jn5 0/1 Running 0 5h29m
machine-config-controller-6fcc66df97-gwfm8 1/1 Running 0 5h36m
machine-config-daemon-78c4k 2/2 Running 0 5h2m
machine-config-daemon-jbhsx 2/2 Running 0 5h42m
machine-config-daemon-rdpfw 2/2 Running 0 4h17m
machine-config-operator-7589466fc6-8bvzj 1/1 Running 0 6h29m
machine-config-server-f5b7h 1/1 Running 0 5h36m
machine-config-server-kpjfx 1/1 Running 0 4h53m
machine-config-server-p9ggm 1/1 Running 0 4h9m
multus-9vlps 1/1 Running 0 4h17m
multus-admission-controller-fmhqb 2/2 Running 0 4h53m
multus-admission-controller-kxkqc 2/2 Running 0 5h29m
multus-admission-controller-xdl8s 2/2 Running 0 4h9m
multus-sl224 1/1 Running 0 5h50m
multus-xm8vb 1/1 Running 0 5h2m
network-operator-75d5b56759-wjtlk 1/1 Running 0 6h30m
catalog-operator-75d9d786b-znlpr 1/1 Running 0 6h29m
olm-operator-6f5d697cfd-vbwtz 1/1 Running 0 6h29m
packageserver-5f5f5745cd-54c2k 1/1 Running 0 32s
packageserver-5f5f5745cd-5x7hc 1/1 Running 0 13s
packageserver-696c5b4cb4-ntkbj 0/1 Terminating 0 5m25s
ovs-866sh 1/1 Running 0 5h2m
ovs-l4ddw 1/1 Running 0 5h49m
ovs-zqcm7 1/1 Running 0 4h17m
sdn-controller-9dvn5 1/1 Running 0 5h49m
sdn-controller-jj77f 1/1 Running 0 5h2m
sdn-controller-q9sfb 1/1 Running 0 4h17m
sdn-g497l 1/1 Running 0 5h2m
sdn-lcc8c 1/1 Running 0 4h17m
sdn-n6w6f 1/1 Running 0 5h49m
service-ca-operator-55ff998c74-hfwdg 1/1 Running 1 6h30m
service-ca-6964f67c9b-mth89 1/1 Running 0 5h39m

[root@bastion ~]# oc get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-4cwzj 7h49m kubernetes.io/kubelet-serving system:node:master1.okd.test1.com Approved,Issued
csr-7cgkn 7h1m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-9vqjh 7h kubernetes.io/kubelet-serving system:node:master2.okd.test1.com Approved,Issued
csr-f4984 6h16m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-kb6h5 74m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-r7759 6h15m kubernetes.io/kubelet-serving system:node:master3.okd.test1.com Approved,Issued
csr-wnbmh 7h49m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued

openshift_install.log

xiaosayishi · 2021-08-16T11:07:32Z

xiaosayishi
Aug 16, 2021
Author

log-bundle:resources

resources.zip

0 replies

xiaosayishi · 2021-08-16T12:41:52Z

xiaosayishi
Aug 16, 2021
Author

The log bundle Connection address ：http://180.97.197.150:180/install/log-bundle-20210816182800.tar.gz

1 reply

rvanderp3 Aug 23, 2021

The kube-apiserver pods appear to be crashlooping due to an inability to reach etcd:


W0816 10:27:10.638793       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://172.16.121.16:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.121.16:2379: connect: connection refused". Reconnecting...
W0816 10:27:11.176845       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://172.16.121.14:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.121.14:2379: connect: connection refused". Reconnecting...
W0816 10:27:11.384694       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://localhost:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp [::1]:2379: connect: connection refused". Reconnecting...
W0816 10:27:12.185677       1 clientconn.go:1208] grpc: addrConn.createTransport failed to connect to {https://172.16.121.15:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.121.15:2379: connect: connection refused". Reconnecting...

etcd static pods appears to be failing to be admitted by the kubelet and explains why the API can't connect:

Aug 16 12:19:35 master1.okd.test1.com hyperkube[875]: W0816 12:19:35.391480     875 predicate.go:112] Failed to admit pod etcd-master1.okd.test1.com_openshift-etcd(9846b9f3a11fc20d5d750e9221b17ca8) - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: cpu, q: 185), ]

The installation appears to be stalling due to a lack of resources. It appears the master nodes have been allocated 2 cores and 8GiB of RAM.

Machine: {NumCores:2 CpuFrequency:2300000 MemoryCapacity:8341319680 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:54e8fe282b0941ea850d960beac046f9 SystemUUID:42231267-abd7-5abf-3d66-e8b199987160 BootID:e00575ec-f076-44ad-9aee-7626156d07d2 Filesystems:[{Device:/dev/sda4 DeviceMajor:8 DeviceMinor:4 Capacity:53138665472 Type:vfs Inodes:25951680 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:43 Capacity:4170657792 Type:vfs Inodes:1018227 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:381549568 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:23 Capacity:4170657792 Type:vfs Inodes:1018227 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:25 Capacity:4170657792 Type:vfs Inodes:1018227 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:26 Capacity:4170657792 Type:vfs Inodes:1018227 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:53687091200 Scheduler:bfq}] NetworkDevices:[{Name:ens192 MacAddress:00:50:56:a3:54:c4 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:8341319680 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[{Size:23068672 Type:Unified Level:3}]} {Id:2 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[{Size:23068672 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}

The minimum requirements are 4 cores and 16GB of RAM for master nodes. Can you try increasing the resources for the master nodes and trying again?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OKD install fail #813

{{title}}

Replies: 2 comments 1 reply

{{title}}

{{title}}

{{title}}

Select a reply

OKD install fail #813

xiaosayishi Aug 16, 2021

Status of the three master nodes：

openshift_install.log

Replies: 2 comments · 1 reply

xiaosayishi Aug 16, 2021 Author

xiaosayishi Aug 16, 2021 Author

rvanderp3 Aug 23, 2021

xiaosayishi
Aug 16, 2021

Replies: 2 comments 1 reply

xiaosayishi
Aug 16, 2021
Author

xiaosayishi
Aug 16, 2021
Author