OKD / Openshift 4.7.0 Bootstrap Failed to start #890

MainMan1998 · 2021-09-23T14:39:07Z

MainMan1998
Sep 23, 2021

Describe the bug
Hello, I'm trying to install okd4 on bare metal server using Fedora Core & Alma linux. The bootstrap is up, but it can't create the cluster.

Openshift installer:

time="2021-09-23T11:55:51+02:00" level=info msg="API v1.20.7-1330+bbbc0799c02950-dirty up"
time="2021-09-23T11:55:51+02:00" level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
time="2021-09-23T12:25:51+02:00" level=info msg="Use the following commands to gather logs from the cluster"
time="2021-09-23T12:25:51+02:00" level=info msg="openshift-install gather bootstrap --help"
time="2021-09-23T12:25:51+02:00" level=fatal msg="failed to wait for bootstrapping to complete: timed out waiting for the condition"

Version:
Vcenter 7 with esxi 7
Almalinux 8
OKD 4.7.0-0.okd-2021-09-19-013247
Fedora Cores 34.20210904.3.0

How reproducible:
I use this guide to install the cluster: https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee
I try to create the cluster more or less 20 attempts. The bootstrap, master, worker vms have the same results.

Log bundle:

Bootstrap vm:

Sep 23 14:03:09 okd4-boostrap systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 14:03:09 okd4-boostrap systemd[1]: bootkube.service: Failed with result 'exit-code'.
Sep 23 14:03:09 okd4-boostrap systemd[1]: bootkube.service: Consumed 19.694s CPU time.
Sep 23 14:03:14 okd4-boostrap systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 2.
Sep 23 14:03:14 okd4-boostrap systemd[1]: Stopped Bootstrap a Kubernetes cluster.
Sep 23 14:03:14 okd4-boostrap systemd[1]: bootkube.service: Consumed 19.694s CPU time.
Sep 23 14:03:14 okd4-boostrap systemd[1]: Started Bootstrap a Kubernetes cluster.

I attached the full log:
bootstrap_okd_log.txt

Master/Worker:

Get "https://api-int.lab.okd.local:22623/config/master": dial tcp: lookup api-int.lab.okd.local on [::1]:53: read udp [::1]:38332->[::1]:53: read: connection failed

Openshift-install

DEBUG OpenShift Installer 4.7.0-0.okd-2021-09-19-013247 
DEBUG Built from commit f44bc61fe81b0f55f56a01a2a183cfd0c8ffb5c9 
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443... 
INFO API v1.20.7-1330+bbbc0799c02950-dirty up     
INFO Waiting up to 30m0s for bootstrapping to complete... 
W0923 15:45:00.504706    3772 reflector.go:424] k8s.io/client-go/tools/watch/informerwatcher.go:146: watch of *v1.ConfigMap ended with: very short watch: k8s.io/client-go/tools/watch/informerwatcher.go:146: Unexpected watch close - watch lasted less than a second and no items received
I0923 15:45:12.792453    3772 trace.go:205] Trace[1369033588]: "Reflector ListAndWatch" name:k8s.io/client-go/tools/watch/informerwatcher.go:146 (23-Sep-2021 15:45:01.539) (total time: 11252ms):
Trace[1369033588]: ---"Objects listed" 11252ms (15:45:00.792)
Trace[1369033588]: [11.252391594s] [11.252391594s] END
INFO Use the following commands to gather logs from the cluster 
INFO openshift-install gather bootstrap --help    
FATAL failed to wait for bootstrapping to complete: timed out waiting for the condition

Configuration File

install-config.yaml

`apiVersion: v1
baseDomain: okd.local
metadata:
name: lab
compute:
- hyperthreading: Enabled
name: worker
replicas: 2
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
networking:
clusterNetwork:
- cidr: 10.128.0.0/14 
hostPrefix: 23
networkType: OpenShiftSDN
serviceNetwork: 
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: 'red hat pull secret'
sshKey: 'ssh-rsa key_centos_vm'

The DNS, HAproxy is correctly configured. Almalinux and fcos have an ip address and the DNS is correctly define. The 22623 port is open on the service vm.

Does anyone have an idea why the bootstrap can't create the cluster ?
Thanks a lot for help.

MainMan1998 · 2021-09-24T19:56:19Z

MainMan1998
Sep 24, 2021
Author

Hello,

After a couple of hours of researching the problem, I discovered that the bootstrap failed to start.

On the beginning of the bootstrap log, I have the following errors:

Sep 24 14:37:53 okd4-bootstrap bootkube.sh[5374]: Failed to create "configmap-admin-kubeconfig-client-ca.yaml" configmaps.v1./admin-kubeconfig-client-ca -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:53 okd4-bootstrap bootkube.sh[5374]: Failed to create "configmap-csr-controller-ca.yaml" configmaps.v1./csr-controller-ca -n openshift-config-managed: namespaces "openshift-config-managed" not found
Sep 24 14:37:54 okd4-bootstrap bootkube.sh[5374]: Failed to create "configmap-kubelet-bootstrap-kubeconfig-ca.yaml" configmaps.v1./kubelet-bootstrap-kubeconfig -n openshift-config-managed: namespaces "openshift-config-managed" not found
Sep 24 14:37:54 okd4-bootstrap bootkube.sh[5374]: Failed to create "configmap-sa-token-signing-certs.yaml" configmaps.v1./sa-token-signing-certs -n openshift-config-managed: namespaces "openshift-config-managed" not found
Sep 24 14:37:54 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-ca-bundle-configmap.yaml" configmaps.v1./etcd-ca-bundle -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:54 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-client-secret.yaml" secrets.v1./etcd-client -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:54 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-metric-client-secret.yaml" secrets.v1./etcd-metric-client -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:55 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-metric-serving-ca-configmap.yaml" configmaps.v1./etcd-metric-serving-ca -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:55 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-metric-signer-secret.yaml" secrets.v1./etcd-metric-signer -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:55 okd4-bootstrap bootkube.sh[5374]: Skipped "etcd-namespace.yaml" namespaces.v1./openshift-etcd -n  as it already exists
Sep 24 14:37:55 okd4-bootstrap bootkube.sh[5374]: Created "etcd-service.yaml" services.v1./etcd -n openshift-etcd
Sep 24 14:37:56 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-serving-ca-configmap.yaml" configmaps.v1./etcd-serving-ca -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:56 okd4-bootstrap bootkube.sh[5374]: Failed to create "etcd-signer-secret.yaml" secrets.v1./etcd-signer -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:56 okd4-bootstrap bootkube.sh[5374]: Failed to create "kube-apiserver-serving-ca-configmap.yaml" configmaps.v1./initial-kube-apiserver-server-ca -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:56 okd4-bootstrap bootkube.sh[5374]: Created "kube-cloud-config.yaml" secrets.v1./kube-cloud-cfg -n kube-system
Sep 24 14:37:56 okd4-bootstrap bootkube.sh[5374]: Created "kube-system-configmap-root-ca.yaml" configmaps.v1./root-ca -n kube-system
Sep 24 14:37:57 okd4-bootstrap bootkube.sh[5374]: Created "machine-config-server-tls-secret.yaml" secrets.v1./machine-config-server-tls -n openshift-machine-config-operator
Sep 24 14:37:57 okd4-bootstrap bootkube.sh[5374]: Failed to create "openshift-config-secret-pull-secret.yaml" secrets.v1./pull-secret -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:57 okd4-bootstrap bootkube.sh[5374]: Failed to create "openshift-install-manifests.yaml" configmaps.v1./openshift-install-manifests -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:57 okd4-bootstrap bootkube.sh[5374]: Created "openshift-kubevirt-infra-namespace.yaml" namespaces.v1./openshift-kubevirt-infra -n
Sep 24 14:37:57 okd4-bootstrap bootkube.sh[5374]: Created "secret-aggregator-client-signer.yaml" secrets.v1./aggregator-client-signer -n openshift-kube-apiserver-operator
Sep 24 14:37:58 okd4-bootstrap bootkube.sh[5374]: Created "secret-bound-sa-token-signing-key.yaml" secrets.v1./next-bound-service-account-signing-key -n openshift-kube-apiserver-operator
Sep 24 14:37:58 okd4-bootstrap bootkube.sh[5374]: Created "secret-control-plane-client-signer.yaml" secrets.v1./kube-control-plane-signer -n openshift-kube-apiserver-operator
Sep 24 14:37:58 okd4-bootstrap bootkube.sh[5374]: Created "secret-csr-signer-signer.yaml" secrets.v1./csr-signer-signer -n openshift-kube-controller-manager-operator
Sep 24 14:37:58 okd4-bootstrap bootkube.sh[5374]: Failed to create "secret-initial-kube-controller-manager-service-account-private-key.yaml" secrets.v1./initial-service-account-private-key -n openshift-config: namespaces "openshift-config" not found
Sep 24 14:37:58 okd4-bootstrap bootkube.sh[5374]: Created "secret-kube-apiserver-to-kubelet-signer.yaml" secrets.v1./kube-apiserver-to-kubelet-signer -n openshift-kube-apiserver-operator
Sep 24 14:37:59 okd4-bootstrap bootkube.sh[5374]: Created "secret-loadbalancer-serving-signer.yaml" secrets.v1./loadbalancer-serving-signer -n openshift-kube-apiserver-operator
Sep 24 14:37:59 okd4-bootstrap bootkube.sh[5374]: Created "secret-localhost-serving-signer.yaml" secrets.v1./localhost-serving-signer -n openshift-kube-apiserver-operator
Sep 24 14:37:59 okd4-bootstrap bootkube.sh[5374]: Created "secret-service-network-serving-signer.yaml" secrets.v1./service-network-serving-signer -n openshift-kube-apiserver-operator

I found on google that the variable machineNetwork specify the MGMT IP scope to the openshift-installer.
I add this variable in the install-config.yaml file but this doesn't resolve the problem.

Log Bundle Update:
Bootstrap VM:
bootstrap_okd_log_update.txt

Openshift-install:

time="2021-09-24T16:16:47+02:00" level=info msg="Consuming Worker Machines from target directory"
time="2021-09-24T16:16:47+02:00" level=debug msg="Purging asset \"Worker Machines\" from disk"
time="2021-09-24T16:16:47+02:00" level=info msg="Consuming Master Machines from target directory"
time="2021-09-24T16:16:47+02:00" level=debug msg="Purging asset \"Master Machines\" from disk"
time="2021-09-24T16:16:47+02:00" level=info msg="Consuming OpenShift Install (Manifests) from target directory"
time="2021-09-24T16:16:47+02:00" level=debug msg="Purging asset \"OpenShift Install (Manifests)\" from disk"
time="2021-09-24T16:16:47+02:00" level=info msg="Consuming Openshift Manifests from target directory"
time="2021-09-24T16:16:47+02:00" level=debug msg="Purging asset \"Openshift Manifests\" from disk"
time="2021-09-24T16:16:47+02:00" level=info msg="Consuming Common Manifests from target directory"
time="2021-09-24T16:16:47+02:00" level=debug msg="Purging asset \"Common Manifests\" from disk"
time="2021-09-24T16:16:47+02:00" level=debug msg="Fetching Metadata..."
time="2021-09-24T16:16:47+02:00" level=debug msg="Loading Metadata..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Loading Cluster ID..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Loading Install Config..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Loading Bootstrap Ignition Config..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Fetching Cluster ID..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Reusing previously-fetched Cluster ID"
time="2021-09-24T16:16:47+02:00" level=debug msg="  Fetching Install Config..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Reusing previously-fetched Install Config"
time="2021-09-24T16:16:47+02:00" level=debug msg="  Fetching Bootstrap Ignition Config..."
time="2021-09-24T16:16:47+02:00" level=debug msg="  Reusing previously-fetched Bootstrap Ignition Config"
time="2021-09-24T16:16:47+02:00" level=debug msg="Generating Metadata..."
time="2021-09-24T16:16:47+02:00" level=info msg="Ignition-Configs created in: install_dir and install_dir/auth"
time="2021-09-24T16:29:50+02:00" level=debug msg="OpenShift Installer 4.7.0-0.okd-2021-09-19-013247"
time="2021-09-24T16:29:50+02:00" level=debug msg="Built from commit f44bc61fe81b0f55f56a01a2a183cfd0c8ffb5c9"
time="2021-09-24T16:29:50+02:00" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443..."
time="2021-09-24T16:30:00+02:00" level=debug msg="Still waiting for the Kubernetes API: an error on the server (\"\") has prevented the request from succeeding"
time="2021-09-24T16:33:01+02:00" level=debug msg="Still waiting for the Kubernetes API: an error on the server (\"\") has prevented the request from succeeding"
time="2021-09-24T16:36:01+02:00" level=debug msg="Still waiting for the Kubernetes API: an error on the server (\"\") has prevented the request from succeeding"
time="2021-09-24T16:37:52+02:00" level=info msg="API v1.20.7-1330+bbbc0799c02950-dirty up"
time="2021-09-24T16:37:52+02:00" level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
time="2021-09-24T17:07:52+02:00" level=info msg="Use the following commands to gather logs from the cluster"
time="2021-09-24T17:07:52+02:00" level=info msg="openshift-install gather bootstrap --help"
time="2021-09-24T17:07:52+02:00" level=fatal msg="failed to wait for bootstrapping to complete: timed out waiting for the condition"

OC-must-gather logs:
oc_must_gather_okd_log.txt

Install-config.yaml:

apiVersion: v1
baseDomain: okd.local
metadata:
  name: lab

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 2

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  networkType: OpenShiftSDN
  serviceNetwork: 
  - 172.30.0.0/16
  machineNetwork:
  - 192.168.30.0/24

platform:
  none: {}

fips: false

pullSecret: 'redhat pull secret' 
sshKey: 'ssh-rsa key_centos_vm.'

Does anyone have an idea why the bootstrap doesn't start ?
Thanks a lot for help.

2 replies

vrutkovs Oct 5, 2021
Maintainer

Bootstrap is waiting for masters to be provisioned and joined as nodes before proceeding.

Get "https://api-int.lab.okd.local:22623/config/master": dial tcp: lookup api-int.lab.okd.local on [::1]:53: read udp [::1]:38332->[::1]:53: read: connection failed

This means the network config received by DHCP doesn't have a valid DNS server, so master can't resolve api-int.lab.okd.local LB address and fetch the remaining ignition from bootstrap node

MainMan1998 Oct 9, 2021
Author

Hi,

Sorry my late answer, I finally found the problem.
It was the fedora core version like this thread: #839
OKD doesn't work with this fedora version : 34.20210904.3.0 but it works with: 34.20210626.3.1

Thanks for help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OKD / Openshift 4.7.0 Bootstrap Failed to start #890

{{title}}

Replies: 1 comment 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

OKD / Openshift 4.7.0 Bootstrap Failed to start #890

MainMan1998 Sep 23, 2021

Replies: 1 comment · 2 replies

MainMan1998 Sep 24, 2021 Author

vrutkovs Oct 5, 2021 Maintainer

MainMan1998 Oct 9, 2021 Author

MainMan1998
Sep 23, 2021

Replies: 1 comment 2 replies

MainMan1998
Sep 24, 2021
Author

vrutkovs Oct 5, 2021
Maintainer

MainMan1998 Oct 9, 2021
Author