release=4.14.0-0.okd-2023-10-28-073550 --- * illegal base64 data at input byte 0 on bootstrap #1784

rlaflamme · 2023-11-05T22:54:43Z

rlaflamme
Nov 5, 2023

Hi,
I successfully upgraded to the latest 4.13.0-0.okd-2023-10-28-065448 but when I try to upgrade to
release 4.14.0-0.okd-2023-10-28-073550, the bootstrap is stuck when starting the bootkube.sh. with this error

bootkube.sh[3227]: * illegal base64 data at input byte 0

Nov 05 22:48:09 okd4-bootstrap podman[3215]: 2023-11-05 22:48:09.730105194 +0000 UTC m=+0.043402740 container remove 5789703e4851350c17d383e231c30261f7037105956bcd2f85921c86c1ae7616 (image=quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c, name=amazing_mayer, io.openshift.release=4.14.0-0.okd-2023-10-28-073550, io.openshift.release.base-image-digest=sha256:068639a8d5ffd7b2e15ee974851875c87b008366fb62a88a32d250ad981c8b47)
Nov 05 22:48:09 okd4-bootstrap bootkube.sh[3227]: error: 1 error occurred:
Nov 05 22:48:09 okd4-bootstrap bootkube.sh[3227]: * illegal base64 data at input byte 0
Nov 05 22:48:09 okd4-bootstrap systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE
Nov 05 22:48:09 okd4-bootstrap systemd[1]: bootkube.service: Failed with result 'exit-code'.
Nov 05 22:48:09 okd4-bootstrap audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=bootkube comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Nov 05 22:48:09 okd4-bootstrap systemd[1]: bootkube.service: Consumed 1.495s CPU time.
Nov 05 22:48:09 okd4-bootstrap systemd[1]: var-lib-containers-storage-overlay.mount: Deactivated successfully.

vrutkovs · 2023-11-06T06:58:01Z

vrutkovs
Nov 6, 2023
Maintainer

Please attach log bundle

when I try to upgrade to release 4.14.0-0.okd-2023-10-28-073550, the bootstrap is stuck

There is no bootstrap during upgrade, do you mean "when I ran a new installation with 4.14"?

3 replies

LorbusChris Nov 6, 2023
Maintainer

@vrutkovs same sometimes occurs with OKD/SCOS: https://redhat-internal.slack.com/archives/CLKF3H5RS/p1698936652588439

rlaflamme Nov 7, 2023
Author

Sorry for the confusion.
It's a new installation, not an upgrade:
Its a Bare Metal installation on Proxmox 8.0.4 (home lab) using terraform:

Memory: 8 GiB
Processors 4 (1 socket, 4 cores) [x86-64-v2]
BIOS: SeaBIOS
Display: Standard VGA (type=std)
Machine Default (i440fx)
SCSI Controller: VirtIO SCSI
Hard Disk (scsi0): local-zfs: vm-200-disk-0,size=120G
Network Device (net0) virtio=, bridge=okdvnet1,mtu-1450,queues=2

Manually gathering logs with SSH access to your host(s)
I attached log files as requested:

bootkube.service: see file in attachment
no logs from "podman logs"
empy file for ctr.log
crio.service: see file in attachment
/var/logs/containers/*: no logs

bootkube.service.log
crio.service.log
pod_logs.log

vrutkovs Nov 7, 2023
Maintainer

Could you add set -x to /usr/local/bin/bootkube.sh on bootstrap machine, restart the service - sudo systemctl restart bootkube and attach the new bootkube.service log?

rlaflamme · 2023-11-07T13:32:19Z

rlaflamme
Nov 7, 2023
Author

I had to struggle with a container storage already in used and finally removed it with 'sudo podman rm --storage before running again the bootkube service.

Observations:
1- I still see the message --- * illegal base64 data at input byte 0 on bootstrap ...
2- Something new..

machine config is up and running I can see it from the LB HAProxy. but still nothing on the frontend.

bootkube.service.log
Now it look likes there is a file already exists issue

(from the log file)

Nov 07 13:26:51 okd4-bootstrap bootkube.sh[175961]: + podman run --quiet --net=host --rm --name cluster-bootstrap --volume /var/opt/openshift:/assets:z --volume /etc/kubernetes:/etc/kubernetes:z quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12>
Nov 07 13:26:51 okd4-bootstrap podman[176564]: 2023-11-07 13:26:51.929109009 +0000 UTC m=+0.029397776 container create cee2db33d67465b41182702f6812d4252f5178956872df04416e741f3af4f288 (image=quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12bd67>
Nov 07 13:26:51 okd4-bootstrap podman[176564]: 2023-11-07 13:26:51.986865739 +0000 UTC m=+0.087154526 container init cee2db33d67465b41182702f6812d4252f5178956872df04416e741f3af4f288 (image=quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12bd676f>
Nov 07 13:26:51 okd4-bootstrap podman[176564]: 2023-11-07 13:26:51.98811959 +0000 UTC m=+0.088408357 container start cee2db33d67465b41182702f6812d4252f5178956872df04416e741f3af4f288 (image=quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12bd676f>
Nov 07 13:26:51 okd4-bootstrap podman[176564]: 2023-11-07 13:26:51.993975491 +0000 UTC m=+0.094264258 container attach cee2db33d67465b41182702f6812d4252f5178956872df04416e741f3af4f288 (image=quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12bd67>
Nov 07 13:26:52 okd4-bootstrap cluster-bootstrap[176575]: Starting temporary bootstrap control plane...
Nov 07 13:26:52 okd4-bootstrap bootkube.sh[176564]: Starting temporary bootstrap control plane...
Nov 07 13:26:52 okd4-bootstrap cluster-bootstrap[176575]: Error: open /etc/kubernetes/manifests/bootstrap-pod.yaml: file exists
Nov 07 13:26:52 okd4-bootstrap cluster-bootstrap[176575]: Tearing down temporary bootstrap control plane...
Nov 07 13:26:52 okd4-bootstrap bootkube.sh[176564]: Error: open /etc/kubernetes/manifests/bootstrap-pod.yaml: file exists
Nov 07 13:26:52 okd4-bootstrap bootkube.sh[176564]: Tearing down temporary bootstrap control plane...
Nov 07 13:26:52 okd4-bootstrap cluster-bootstrap[176575]: Error: open /etc/kubernetes/manifests/bootstrap-pod.yaml: file exists
Nov 07 13:26:52 okd4-bootstrap bootkube.sh[176564]: Error: open /etc/kubernetes/manifests/bootstrap-pod.yaml: file exists

1 reply

vrutkovs Nov 7, 2023
Maintainer

Nov 07 13:15:22 okd4-bootstrap bootkube.sh[156414]: ++ oc adm release info -o 'jsonpath={.metadata.version}' '' quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c
Nov 07 13:15:22 okd4-bootstrap bootkube.sh[156414]: error: 1 error occurred:
Nov 07 13:15:22 okd4-bootstrap bootkube.sh[156414]:         * illegal base64 data at input byte 0

That looks like an oc bug

rlaflamme · 2023-11-13T23:00:01Z

rlaflamme
Nov 13, 2023
Author

Unfortunately, I am still having the issue with 4.14.0-0.okd-2023-11-12-042703

0 replies

YOKurnaz · 2023-11-15T07:39:05Z

YOKurnaz
Nov 15, 2023

Hi, I am having the same problem on my lab environment at a fresh new installation. I even tried 4.14.0-0.okd-2023-11-14-101924. Should I open a bug?

0 replies

YOKurnaz · 2023-11-15T07:55:58Z

YOKurnaz
Nov 15, 2023

okd4_illegal_base64_data_error.txt

When I run "oc adm release info -o 'jsonpath={.metadata.version}' '' quay.io/openshift/okd@sha256:72d40c51e7c4d1b9c31e9b0d276d045f1b2b93def5ecee49186df856d40bcb5c" command (where I got the error) I was able to get proper reply/version.

[core@okdbs ~]$ oc adm release info -o 'jsonpath={.metadata.version}' '' quay.io/openshift/okd@sha256:72d40c51e7c4d1b9c31e9b0d276d045f1b2b93def5ecee49186df856d40bcb5c
4.14.0-0.okd-2023-11-14-101924[core@okdbs ~]$
[core@okdbs ~]$

7 replies

YOKurnaz Nov 15, 2023

Right now I commented out the VERSION variable line and assign the output of "4.14.0-0.okd-2023-11-14-101924" to VERSION by hand. (Waiting for the result.)

vrutkovs Nov 15, 2023
Maintainer

hmm, fake pullsecret should work.

Are you using any mirror? Perhaps its related to recent quay.io instability, but its hard to tell without a log bundle

YOKurnaz Nov 15, 2023

I think there is a problem with coreos or commands to get version number. What have I done is change the /usr/local/bin/bootkube.sh file as below:

Get the command for version from bootkube.sh script (in my situation the command was -> oc adm release info -o 'jsonpath={.metadata.version}' '' quay.io/openshift/okd@sha256:72d40c51e7c4d1b9c31e9b0d276d045f1b2b93def5ecee49186df856d40bcb5c)
Run the full command on shell to get version (my output was like above -> 4.14.0-0.okd-2023-11-14-101924[core@okdbs ~]$ )
At line 84 VERSION variable is assigned and we were having error at parsing version. So I comment out the line (put # in the beginning of line) and added the following line below

VERSION="4.14.0-0.okd-2023-11-14-101924"
Restarted bootkube service "sudo systemctl restart bootkube"

These solved my problem and now I am waiting for controller03 to start. But this will be revert back if bootstrap server will restart. I hope this will be useful.

KIgnas Nov 15, 2023

This fake pullsecret does not work since bar is invalid base64. pullSecret: '{"auths":{"fake":{"auth": ""}}}' worked for us instead

vrutkovs Nov 15, 2023
Maintainer

oh! yeah, I remember fixing README to use {"auths":{"fake":{"auth":"aWQ6cGFzcwo="}} on a related case. Anyone seen if docs list "bar" in plain text?

rlaflamme · 2023-11-19T15:43:54Z

rlaflamme
Nov 19, 2023
Author

Nice catch!! It was a pullsecret issue. I had too the value "bar". Until 4.13 always did the job. FWI the source of the bar value, the one I used in my configuration come from here --> https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee and more specifically here --> https://github.com/cragr/okd4_files/blob/master/install-config.yaml

Finally got working
Thanks you @vrutkovs @KIgnas @LorbusChris @YOKurnaz !

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

release=4.14.0-0.okd-2023-10-28-073550 --- * illegal base64 data at input byte 0 on bootstrap #1784

{{title}}

Replies: 6 comments 11 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

release=4.14.0-0.okd-2023-10-28-073550 --- * illegal base64 data at input byte 0 on bootstrap #1784

rlaflamme Nov 5, 2023

Replies: 6 comments · 11 replies

vrutkovs Nov 6, 2023 Maintainer

LorbusChris Nov 6, 2023 Maintainer

rlaflamme Nov 7, 2023 Author

vrutkovs Nov 7, 2023 Maintainer

rlaflamme Nov 7, 2023 Author

vrutkovs Nov 7, 2023 Maintainer

rlaflamme Nov 13, 2023 Author

YOKurnaz Nov 15, 2023

YOKurnaz Nov 15, 2023

YOKurnaz Nov 15, 2023

vrutkovs Nov 15, 2023 Maintainer

YOKurnaz Nov 15, 2023

KIgnas Nov 15, 2023

vrutkovs Nov 15, 2023 Maintainer

rlaflamme Nov 19, 2023 Author

rlaflamme
Nov 5, 2023

Replies: 6 comments 11 replies

vrutkovs
Nov 6, 2023
Maintainer

LorbusChris Nov 6, 2023
Maintainer

rlaflamme Nov 7, 2023
Author

vrutkovs Nov 7, 2023
Maintainer

rlaflamme
Nov 7, 2023
Author

vrutkovs Nov 7, 2023
Maintainer

rlaflamme
Nov 13, 2023
Author

YOKurnaz
Nov 15, 2023

YOKurnaz
Nov 15, 2023

vrutkovs Nov 15, 2023
Maintainer

vrutkovs Nov 15, 2023
Maintainer

rlaflamme
Nov 19, 2023
Author