OKD4 : Error: unhealthy cluster #1735

balioune · 2023-09-12T17:13:54Z

balioune
Sep 12, 2023

Describe the bug
Hi,
I'm trying to deploy ODK4 but the bootstrap process fails at installation of etcd (service bootkube.service).
The etdc container is lanched during 5 seconds, removed and then restarte again with the log below.

This is the podman command executed from /usr/local/bin/bootkube.sh

    podman run --quiet --net=host --rm 
        --name etcdctl \
        --env ETCDCTL_API=3 \
        --volume /opt/openshift/tls:/opt/openshift/tls:ro,z \
        --entrypoint etcdctl \
        "${MACHINE_CONFIG_ETCD_IMAGE}" \
        --dial-timeout=10m \
        --cacert=/opt/openshift/tls/etcd-ca-bundle.crt \
        --cert=/opt/openshift/tls/etcd-client.crt \
        --key=/opt/openshift/tls/etcd-client.key \
        --endpoints="${ETCD_ENDPOINTS}" \
        endpoint health

Please any suggestion to solve this issue ?
BR,

OKD Version
OKD version: 4.13.0-0.okd-2023-07-23-051208

Podman version
Client: Podman Engine
Version: 4.6.1
API Version: 4.6.1
Go Version: go1.20.7
Built: Thu Aug 10 22:07:53 2023
OS/Arch: linux/amd64

FCOS Version
[core@host-10-253-235-60 ~]$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38.20230819.3.0 (CoreOS)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora CoreOS 38.20230819.3.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='38.20230819.3.0'

How reproducible

Log bundle
Sep 12 16:59:54 host-10-253-235-60 etcdctl[29990]: {"level":"warn","ts":"2023-09-12T16:59:54.372Z","logger":"client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001ba000/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:2379: connect: connection refused""}
Sep 12 16:59:54 host-10-253-235-60 etcdctl[29990]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Sep 12 16:59:54 host-10-253-235-60 bootkube.sh[29973]: {"level":"warn","ts":"2023-09-12T16:59:54.372Z","logger":"client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001ba000/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:2379: connect: connection refused""}
Sep 12 16:59:54 host-10-253-235-60 bootkube.sh[29973]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Sep 12 16:59:54 host-10-253-235-60 bootkube.sh[29973]: Error: unhealthy cluster
Sep 12 16:59:54 host-10-253-235-60 etcdctl[29990]: Error: unhealthy cluster
Sep 12 16:59:54 host-10-253-235-60 podman[29973]: 2023-09-12 16:59:54.378029957 +0000 UTC m=+5.187958005 container died 58f168841c1a79f8c3d1e5719e81e50f67673a208546a93dc3c53a21f0629429 (image=quay.io/openshift/okd-content@sha256:db91912b0e3c7c3ddef645b6bb057f25df5bf67b2ebcc0ddd64aa9fd3657a969, name=etcdctl, io.openshift.build.source-location=https://github.com/openshift/etcd, io.openshift.ci.from.base=sha256:c788c1fe9ad10925bff3935884a656c025293f80e06638c6fe2ac00ee9c9df7c, vendor=Red Hat, Inc., io.k8s.description=etcd is a distributed key-value store which stores the persistent master state for Kubernetes and OpenShift., summary=Provides the latest release of the Red Hat Extended Life Base Image., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.13.0-202306070816.p0.g0465934.assembly.stream, build-date=2023-06-07T13:41:57, io.openshift.build.namespace=, name=openshift/ose-base, io.openshift.build.commit.id=f70da9d78221bc3e6bf8ac14c0c4ecc106f4f57d, com.redhat.build-host=cpt-1001.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.build.commit.message=, io.k8s.display-name=etcd server, io.openshift.build.commit.author=, io.openshift.build.commit.url=openshift/images@0465934, io.openshift.build.source-context-dir=, vcs-url=https://github.com/openshift/etcd, description=This is the base image from which all OpenShift Container Platform images inherit., io.buildah.version=1.29.0, distribution-scope=public, version=v4.13.0, io.openshift.build.commit.date=, maintainer=Sam Batschelet [email protected], io.openshift.expose-services=, io.openshift.maintainer.component=Release, com.redhat.license_terms=https://www.redhat.com/agreements, release=202306070816.p0.g0465934.assembly.stream, io.openshift.tags=openshift,base, License=GPLv2+, io.openshift.maintainer.project=OCPBUGS, vcs-ref=f70da9d78221bc3e6bf8ac14c0c4ecc106f4f57d, io.openshift.build.name=, io.openshift.build.commit.ref=openshift-4.13, com.redhat.component=openshift-enterprise-base-container)
Sep 12 16:59:54 host-10-253-235-60 podman[30006]: 2023-09-12 16:59:54.475050294 +0000 UTC m=+0.082769545 container remove 58f168841c1a79f8c3d1e5719e81e50f67673a208546a93dc3c53a21f0629429 (image=quay.io/openshift/okd-content@sha256:db91912b0e3c7c3ddef645b6bb057f25df5bf67b2ebcc0ddd64aa9fd3657a969, name=etcdctl, vcs-ref=f70da9d78221bc3e6bf8ac14c0c4ecc106f4f57d, io.k8s.display-name=etcd server, vcs-url=https://github.com/openshift/etcd, com.redhat.license_terms=https://www.redhat.com/agreements, io.openshift.tags=openshift,base, io.openshift.build.source-context-dir=, License=GPLv2+, io.openshift.maintainer.component=Release, io.openshift.build.commit.ref=openshift-4.13, com.redhat.build-host=cpt-1001.osbs.prod.upshift.rdu2.redhat.com, summary=Provides the latest release of the Red Hat Extended Life Base Image., release=202306070816.p0.g0465934.assembly.stream, io.openshift.build.namespace=, build-date=2023-06-07T13:41:57, io.openshift.build.commit.author=, io.buildah.version=1.29.0, name=openshift/ose-base, vendor=Red Hat, Inc., io.openshift.ci.from.base=sha256:c788c1fe9ad10925bff3935884a656c025293f80e06638c6fe2ac00ee9c9df7c, maintainer=Sam Batschelet [email protected], url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-base/images/v4.13.0-202306070816.p0.g0465934.assembly.stream, description=This is the base image from which all OpenShift Container Platform images inherit., com.redhat.component=openshift-enterprise-base-container, io.openshift.build.name=, distribution-scope=public, io.openshift.build.commit.url=openshift/images@0465934, io.openshift.build.commit.message=, io.openshift.build.commit.id=f70da9d78221bc3e6bf8ac14c0c4ecc106f4f57d, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, io.openshift.maintainer.project=OCPBUGS, io.k8s.description=etcd is a distributed key-value store which stores the persistent master state for Kubernetes and OpenShift., version=v4.13.0, io.openshift.build.commit.date=, io.openshift.build.source-location=https://github.com/openshift/etcd)
Sep 12 16:59:54 host-10-253-235-60 bootkube.sh[23703]: etcdctl failed. Retrying in 5 seconds...

Answered by vrutkovs

Sep 14, 2023

W0913 20:08:06.283724       1 reflector.go:424] github.com/openshift/cluster-bootstrap/pkg/start/status.go:127: failed to list *v1.Pod: Get "https://localhost:6443/api/v1/pods": x509: certificate has expired or is not yet valid: current time 2023-09-13T20:08:06Z is after 2023-09-13T16:46:34Z

Apparently certs generated by the installer have expired - these are valid for 24 hours only. You'd need to reprovision nodes again using new Ignition files

View full answer

vrutkovs · 2023-09-12T18:22:29Z

vrutkovs
Sep 12, 2023
Maintainer

See this instruction about log bundle archive

2 replies

balioune Sep 12, 2023
Author

thanks @vrutkovs.
In gathers file is seems the bootstrap does not detect masters but the FQDN api-int.apps.domain is up and respond to ping

log-bundle-20230912164045.tar.gz

vrutkovs Sep 13, 2023
Maintainer

release image pivot is failing:

  UNIT                        LOAD   ACTIVE SUB    DESCRIPTION
* release-image-pivot.service loaded failed failed Pivot bootstrap to the OpenShift Release Image

Image cannot be pulled from the mirror:

Sep 12 20:34:55 host-10-253-235-60 rpm-ostree[9821]: Txn Rebase on /org/projectatomic/rpmostree1/fedora_coreos failed: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: (Mirrors also failed: [openshift.artifactory.mycloud.intrabpce.fr/openshift/okd-content@sha256:91667082a219d959f55edddfa684e5113a52f4028abb5b479a557abe14714cad: getting username and password: 1 error occurred:
                                                             * reading JSON file "/run/containers/62011/auth.json": open /run/containers/62011/auth.json: permission denied
                                                     
                                                     ]): quay.io/openshift/okd-content@sha256:91667082a219d959f55edddfa684e5113a52f4028abb5b479a557abe14714cad: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io: no such host

See #1500

balioune · 2023-09-13T14:27:52Z

balioune
Sep 13, 2023
Author

Thanks @vrutkovs I solve the issue with these steps

sudo mkdir /run/containers/62011
sudo cp /root/.docker/config.json /run/containers/62011/auth.json
sudo chmod a+r /run/containers/62011/auth.json
sudo chmod a+rx /run/containers
sudo systemctl restart release-image-pivot.service

Right now the kube-apiserver is not running not running on port 6443 on the master.
Do you already deal with this error ? (I put some logs in the attached)
BR,

kube-apiserver.log

1 reply

vrutkovs Sep 13, 2023
Maintainer

It looks like running to me. Please collect another log bundle for more info

balioune · 2023-09-13T20:24:30Z

balioune
Sep 13, 2023
Author

@vrutkovs you can find the log in the attached.

We can also see the the kube-api-server container is down.

[core@host-10-253-235-60 ~]$ sudo crictl ps -a
CONTAINER           IMAGE                                                                                                   CREATED              STATE               NAME                             ATTEMPT             POD ID              POD
1534cc5e1030b       1ec692bf111460c8ca1f5642ebbd2bc04a4c409c8928145c0882b76932f8d479                                        12 seconds ago       Running             kube-apiserver                   7                   8d81c1ae56c0d       bootstrap-kube-apiserver-host-10-253-235-60
dfb406cecd97a       quay.io/openshift/okd@sha256:f0e95c44ff7307fc4b3927ab2f1a9d30c2688f4e4ad3e8d702f04b1171bf84c6           52 seconds ago       Exited              cluster-version-operator         7                   2c74bc5a32f7e       bootstrap-cluster-version-operator-host-10-253-235-60
91d2cab0a7abf       12792c14450dcac749e5968dcb0a7ef237960e179ea3626413f4c404aeda2586                                        About a minute ago   Exited              cluster-policy-controller        9                   24f787d4ff5ac       bootstrap-kube-controller-manager-host-10-253-235-60
3f5421e9df208       1ec692bf111460c8ca1f5642ebbd2bc04a4c409c8928145c0882b76932f8d479                                        4 minutes ago        Exited              kube-controller-manager          7                   24f787d4ff5ac       bootstrap-kube-controller-manager-host-10-253-235-60
78598dbedf33f       1ec692bf111460c8ca1f5642ebbd2bc04a4c409c8928145c0882b76932f8d479                                        5 minutes ago        Exited              kube-apiserver                   6                   8d81c1ae56c0d       bootstrap-kube-apiserver-host-10-253-235-60
aa424c2588949       6e996ba513f20896fa5a39e39ac2f2093d23500071c5e98fe5e502d548d7aff2                                        14 minutes ago       Running             kube-apiserver-insecure-readyz   0                   8d81c1ae56c0d       bootstrap-kube-apiserver-host-10-253-235-60
95ac671f57f93       quay.io/openshift/okd-content@sha256:6c020d986a4b6b51ed36b9d36dd58cbda857193dff5ffa4589e6e0a57ea47bcf   14 minutes ago       Running             kube-scheduler                   0                   fa430634373a9       bootstrap-kube-scheduler-host-10-253-235-60
4084a961a8bc6       quay.io/openshift/okd-content@sha256:6c020d986a4b6b51ed36b9d36dd58cbda857193dff5ffa4589e6e0a57ea47bcf   14 minutes ago       Exited              setup                            0                   8d81c1ae56c0d       bootstrap-kube-apiserver-host-10-253-235-60
c83e309e364d6       916218223532f2d035fa637995f1eba461831607d9eadd7d2460b3c353646007                                        15 minutes ago       Running             cloud-credential-operator        0                   aa498b9785cc4       cloud-credential-operator-host-10-253-235-60
ce7149c96e894       b4f5b515ff3096dc197b933071faf9498882bfcaf34e63c83f5a54fd1fa58da7                                        15 minutes ago       Running             machine-config-server            0                   e47c87e4a5aee       bootstrap-machine-config-operator-host-10-253-235-60
8f098e50605da       b4f5b515ff3096dc197b933071faf9498882bfcaf34e63c83f5a54fd1fa58da7                                        15 minutes ago       Exited              machine-config-controller        0                   e47c87e4a5aee       bootstrap-machine-config-operator-host-10-253-235-60
eaeb721a45e74       fd9ca1d5e560b30145557a492671dbe43049ce23995b3d6c685d7b4c31c54351                                        15 minutes ago       Running             etcd                             0                   d91aef8ea86c2       etcd-bootstrap-member-host-10-253-235-60
3aaf6383372c0       quay.io/openshift/okd-content@sha256:db91912b0e3c7c3ddef645b6bb057f25df5bf67b2ebcc0ddd64aa9fd3657a969   15 minutes ago       Running             etcdctl                          0                   d91aef8ea86c2       etcd-bootstrap-member-host-10-253-235-60

log-bundle-20230913161556.tar.gz

1 reply

vrutkovs Sep 14, 2023
Maintainer

W0913 20:08:06.283724       1 reflector.go:424] github.com/openshift/cluster-bootstrap/pkg/start/status.go:127: failed to list *v1.Pod: Get "https://localhost:6443/api/v1/pods": x509: certificate has expired or is not yet valid: current time 2023-09-13T20:08:06Z is after 2023-09-13T16:46:34Z

Apparently certs generated by the installer have expired - these are valid for 24 hours only. You'd need to reprovision nodes again using new Ignition files

Answer selected by JaimeMagiera

balioune · 2023-09-14T10:26:17Z

balioune
Sep 14, 2023
Author

Thanks so much @vrutkovs I removed the certificat in the install-config.yaml file.
Now the master and node are running, I can use command oc and kubectl on the cluster.

NAME                 STATUS   ROLES                  AGE     VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION          CONTAINER-RUNTIME
host-10-253-235-55   Ready    control-plane,master   58m     v1.26.7+0ef5eae   10.253.235.55   <none>        Fedora CoreOS 38.20230806.3.0   6.4.7-200.fc38.x86_64   cri-o://1.26.4
host-10-253-235-57   Ready    worker                 7m12s   v1.26.7+0ef5eae   10.253.235.57   <none>        Fedora CoreOS 38.20230806.3.0   6.4.7-200.fc38.x86_64   cri-o://1.26.4

I can not access to the web console via https://console-openshift-console.apps.mydomain. Please how to enable it ?
it seems that its pod is running!

openshift-console-operator                         console-operator-5c87b99879-tztb9                            2/2     Running                  0             54m     10.128.0.63     host-10-253-235-55   <none>           <none>
openshift-console                                  downloads-59c5768c4-bzgxg                                    1/1     Running                  0             53m     10.128.0.66     host-10-253-235-55   <none>           <none>

I try to edit the yaml conf : oc edit console.config.openshift.io cluster and and the following but the seems the edit is not taken into account.

status:
  consoleURL: https://console-openshift-console.apps.mydomain

I see also some pods in the kube-apiserver namesapace are not running

$ ./kubectl get pods -n openshift-kube-apiserver
NAME                                READY   STATUS      RESTARTS   AGE
installer-3-host-10-253-235-55      0/1     Error       0          138m
installer-4-host-10-253-235-55      0/1     Completed   0          134m
installer-5-host-10-253-235-55      0/1     Completed   0          132m
installer-7-host-10-253-235-55      0/1     Completed   0          129m
kube-apiserver-host-10-253-235-55   5/5     Running     0          128m

./oc get po -A | grep -v Running | grep -v Completed

NAMESPACE                                          NAME                                                         READY   STATUS
                   RESTARTS       AGE
openshift-ingress                                  router-default-bbb4fc6b6-rtgrd                               0/1     Pending                  0              151m
openshift-kube-apiserver                           installer-3-host-10-253-235-55                               0/1     Error                    0              149m
openshift-marketplace                              community-operators-cjblf                                    0/1     ImagePullBackOff         0              151m
openshift-marketplace                              community-operators-ft8vj                                    0/1     ImagePullBackOff         0              142m
openshift-network-diagnostics                      network-check-source-95f855b4-252g9                          0/1     ContainerStatusUnknown   0              93m
openshift-network-diagnostics                      network-check-source-95f855b4-2mplk                          0/1     ContainerStatusUnknown   0              14m
openshift-network-diagnostics                      network-check-source-95f855b4-2trlp                          0/1     ContainerStatusUnknown   0              93mopenshift-network-diagnostics                      network-check-source-95f855b4-2vs5h                          0/1     ContainerStatusUnknown   0              14m
openshift-network-diagnostics                      network-check-source-95f855b4-48kjg                          0/1     ContainerStatusUnknown   0              93m
openshift-network-diagnostics                      network-check-source-95f855b4-4lft6                          0/1     ContainerStatusUnknown   0              93mopenshift-network-diagnostics                      network-check-source-95f855b4-4npth                          0/1     ContainerStatusUnknown   0              5m5s
openshift-network-diagnostics                      network-check-source-95f855b4-4tksl                          0/1     ContainerStatusUnknown   0              4m59s
openshift-network-diagnostics                      network-check-source-95f855b4-55snr                          0/1     ContainerStatusUnknown   0              80m
openshift-network-diagnostics                      network-check-source-95f855b4-65lvr                          0/1     ContainerStatusUnknown   0              80m
openshift-network-diagnostics                      network-check-source-95f855b4-6kzqj                          0/1     Contai

[1] https://docs.okd.io/latest/web_console/configuring-web-console.html
BR,

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OKD4 : Error: unhealthy cluster #1735

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

OKD4 : Error: unhealthy cluster #1735

balioune Sep 12, 2023

Replies: 4 comments · 4 replies

vrutkovs Sep 12, 2023 Maintainer

balioune Sep 12, 2023 Author

vrutkovs Sep 13, 2023 Maintainer

balioune Sep 13, 2023 Author

vrutkovs Sep 13, 2023 Maintainer

balioune Sep 13, 2023 Author

vrutkovs Sep 14, 2023 Maintainer

balioune Sep 14, 2023 Author

balioune
Sep 12, 2023

Replies: 4 comments 4 replies

vrutkovs
Sep 12, 2023
Maintainer

balioune Sep 12, 2023
Author

vrutkovs Sep 13, 2023
Maintainer

balioune
Sep 13, 2023
Author

vrutkovs Sep 13, 2023
Maintainer

balioune
Sep 13, 2023
Author

vrutkovs Sep 14, 2023
Maintainer

balioune
Sep 14, 2023
Author