OS update error during latest update 4.14.0-0.okd-2023-11-12-042703 #1794

llomgui · 2023-11-13T10:51:38Z

llomgui
Nov 13, 2023

Describe the bug
The first nodes Master and Worker tried to do their OS upgrade post OKD update.
After more than 1 hour, they are still Not Ready.

Version
Update from 4.14.0-0.okd-2023-10-28-073550 to 4.14.0-0.okd-2023-11-12-042703

Log bundle
According to journalctl command, there is an issue with the memory cache.

Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: Starting kubelet.service - Kubernetes Kubelet...
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org crio[32735]: time="2023-11-13 10:41:15.571394176Z" level=info msg="Starting CRI-O, version: 1.28.2, git: unknown(clean)"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: crio-wipe.service: Deactivated successfully.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: Finished crio-wipe.service - CRI-O Auto Update Script.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=crio-wipe comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kernel: audit: type=1130 audit(1699872075.578:1411): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=crio-wipe comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kernel: audit: type=1131 audit(1699872075.578:1412): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=crio-wipe comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=crio-wipe comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: machine-config-daemon-pull.service - Machine Config Daemon Pull was skipped because of an unmet condition check (ConditionPathExists=/etc/ignition-machine-config-encapsulated.json).
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: machine-config-daemon-firstboot.service - Machine Config Daemon Firstboot was skipped because of an unmet condition check (ConditionPathExists=/etc/ignition-machine-config-encapsulated.json).
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --cloud-provider has been deprecated, will be removed in 1.25 or later, in favor of removing cloud provider code from Kubelet.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --cloud-config has been deprecated, will be removed in 1.25 or later, in favor of removing cloud provider code from Kubelet.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --register-with-taints has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.600010   32747 server.go:199] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.600992   32747 feature_gate.go:227] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601011   32747 feature_gate.go:227] unrecognized feature gate: AzureWorkloadIdentity
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601015   32747 feature_gate.go:227] unrecognized feature gate: AutomatedEtcdBackup
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601019   32747 feature_gate.go:227] unrecognized feature gate: SigstoreImageVerification
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601023   32747 feature_gate.go:227] unrecognized feature gate: GatewayAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601028   32747 feature_gate.go:227] unrecognized feature gate: AlibabaPlatform
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601034   32747 feature_gate.go:227] unrecognized feature gate: GCPLabelsTags
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601038   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderExternal
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601042   32747 feature_gate.go:227] unrecognized feature gate: PrivateHostedZoneAWS
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601046   32747 feature_gate.go:227] unrecognized feature gate: CSIDriverSharedResource
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601050   32747 feature_gate.go:227] unrecognized feature gate: RouteExternalCertificate
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601053   32747 feature_gate.go:227] unrecognized feature gate: AdminNetworkPolicy
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601057   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601060   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIProviderOpenStack
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601064   32747 feature_gate.go:227] unrecognized feature gate: VSphereStaticIPs
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601067   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderAzure
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601071   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderGCP
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601074   32747 feature_gate.go:227] unrecognized feature gate: InsightsConfigAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.601078   32747 feature_gate.go:227] unrecognized feature gate: BuildCSIVolumes
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602365   32747 feature_gate.go:227] unrecognized feature gate: PrivateHostedZoneAWS
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602374   32747 feature_gate.go:227] unrecognized feature gate: CSIDriverSharedResource
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602381   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderExternal
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602389   32747 feature_gate.go:227] unrecognized feature gate: AdminNetworkPolicy
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602395   32747 feature_gate.go:227] unrecognized feature gate: RouteExternalCertificate
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602402   32747 feature_gate.go:227] unrecognized feature gate: VSphereStaticIPs
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602408   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderAzure
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602414   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderGCP
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602425   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602436   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIProviderOpenStack
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602443   32747 feature_gate.go:227] unrecognized feature gate: BuildCSIVolumes
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602449   32747 feature_gate.go:227] unrecognized feature gate: InsightsConfigAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602456   32747 feature_gate.go:227] unrecognized feature gate: AzureWorkloadIdentity
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602475   32747 feature_gate.go:227] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602482   32747 feature_gate.go:227] unrecognized feature gate: AutomatedEtcdBackup
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602489   32747 feature_gate.go:227] unrecognized feature gate: SigstoreImageVerification
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602495   32747 feature_gate.go:227] unrecognized feature gate: AlibabaPlatform
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602505   32747 feature_gate.go:227] unrecognized feature gate: GatewayAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.602512   32747 feature_gate.go:227] unrecognized feature gate: GCPLabelsTags
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.602519   32747 feature_gate.go:250] feature gates: &{map[AdmissionWebhookMatchConditions:false CloudDualStackNodeIPs:true DynamicResourceAllocation:false EventedPLEG:false MaxUnavailableStatefulSet:false NodeSwap:false RetroactiveDefaultStorageClass:false ValidatingAdmissionPolicy:false]}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.605371   32747 server.go:415] "Kubelet version" kubeletVersion="v1.27.6+b49f9d1"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.605388   32747 server.go:417] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605424   32747 feature_gate.go:227] unrecognized feature gate: GCPLabelsTags
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605429   32747 feature_gate.go:227] unrecognized feature gate: CSIDriverSharedResource
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605433   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderExternal
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605437   32747 feature_gate.go:227] unrecognized feature gate: PrivateHostedZoneAWS
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605441   32747 feature_gate.go:227] unrecognized feature gate: AdminNetworkPolicy
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605445   32747 feature_gate.go:227] unrecognized feature gate: RouteExternalCertificate
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605449   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderAzure
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605452   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderGCP
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605456   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605478   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIProviderOpenStack
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605485   32747 feature_gate.go:227] unrecognized feature gate: VSphereStaticIPs
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605493   32747 feature_gate.go:227] unrecognized feature gate: BuildCSIVolumes
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605497   32747 feature_gate.go:227] unrecognized feature gate: InsightsConfigAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605500   32747 feature_gate.go:227] unrecognized feature gate: AzureWorkloadIdentity
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605504   32747 feature_gate.go:227] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605508   32747 feature_gate.go:227] unrecognized feature gate: AutomatedEtcdBackup
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605512   32747 feature_gate.go:227] unrecognized feature gate: SigstoreImageVerification
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605516   32747 feature_gate.go:227] unrecognized feature gate: AlibabaPlatform
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605519   32747 feature_gate.go:227] unrecognized feature gate: GatewayAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.605524   32747 feature_gate.go:250] feature gates: &{map[AdmissionWebhookMatchConditions:false CloudDualStackNodeIPs:true DynamicResourceAllocation:false EventedPLEG:false MaxUnavailableStatefulSet:false NodeSwap:false RetroactiveDefaultStorageClass:false ValidatingAdmissionPolicy:false]}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605593   32747 feature_gate.go:227] unrecognized feature gate: AzureWorkloadIdentity
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605598   32747 feature_gate.go:227] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605602   32747 feature_gate.go:227] unrecognized feature gate: AutomatedEtcdBackup
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605605   32747 feature_gate.go:227] unrecognized feature gate: SigstoreImageVerification
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605608   32747 feature_gate.go:227] unrecognized feature gate: AlibabaPlatform
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605613   32747 feature_gate.go:227] unrecognized feature gate: GatewayAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605617   32747 feature_gate.go:227] unrecognized feature gate: GCPLabelsTags
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605621   32747 feature_gate.go:227] unrecognized feature gate: PrivateHostedZoneAWS
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605624   32747 feature_gate.go:227] unrecognized feature gate: CSIDriverSharedResource
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605629   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderExternal
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605633   32747 feature_gate.go:227] unrecognized feature gate: AdminNetworkPolicy
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605637   32747 feature_gate.go:227] unrecognized feature gate: RouteExternalCertificate
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605640   32747 feature_gate.go:227] unrecognized feature gate: VSphereStaticIPs
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605644   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderAzure
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605647   32747 feature_gate.go:227] unrecognized feature gate: ExternalCloudProviderGCP
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605650   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605655   32747 feature_gate.go:227] unrecognized feature gate: MachineAPIProviderOpenStack
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605659   32747 feature_gate.go:227] unrecognized feature gate: BuildCSIVolumes
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605663   32747 feature_gate.go:227] unrecognized feature gate: InsightsConfigAPI
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.605668   32747 feature_gate.go:250] feature gates: &{map[AdmissionWebhookMatchConditions:false CloudDualStackNodeIPs:true DynamicResourceAllocation:false EventedPLEG:false MaxUnavailableStatefulSet:false NodeSwap:false RetroactiveDefaultStorageClass:false ValidatingAdmissionPolicy:false]}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.605784   32747 plugins.go:130] WARNING: gce built-in cloud provider is now deprecated. The GCE provider is deprecated and will be removed in a future release. Please use https://github.com/kubernetes/cloud-provider-gcp
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.605959   32747 gce.go:292] Using GCE provider config &{Global:{TokenURL: TokenBody: ProjectID:prj-myorg-okd-change-xy45 NetworkProjectID:prj-myorg-3aa4 NetworkName:vpc-myorg-shared SubnetworkName:snet-myorg-okd-change-euwe6 SecondaryRangeName: NodeTags:[okd-ci-ljhg6-master okd-ci-ljhg6-worker] NodeInstancePrefix:okd-ci-ljhg6 Regional:true Multizone:true ExternalInstanceGroupsPrefix:okd-ci-ljhg6 APIEndpoint: ContainerAPIEndpoint: LocalZone: AlphaFeatures:[]}}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.607661   32747 gce.go:942] Using existing Token Source &oauth2.reuseTokenSource{new:google.computeSource{account:"", scopes:[]string(nil)}, mu:sync.Mutex{state:0, sema:0x0}, t:(*oauth2.Token)(nil)}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.887791   32747 gce.go:512] managing multiple zones: [europe-west6-a europe-west6-b europe-west6-c]
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.887859   32747 server.go:557] "Successfully initialized cloud provider" cloudProvider="gce" cloudConfigFile="/etc/kubernetes/cloud.conf"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.887894   32747 server.go:1011] "Cloud provider determined current node" nodeName="okd-ci-ljhg6-master-a.domain.org"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.887906   32747 server.go:843] "Client rotation is on, will bootstrap in background"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890111   32747 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890191   32747 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890523   32747 server.go:900] "Starting client certificate rotation"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890549   32747 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Certificate rotation is enabled
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890793   32747 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Certificate expiration is 2023-11-17 22:16:14 +0000 UTC, rotation deadline is 2023-11-15 02:50:45.884005484 +0000 UTC
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.890946   32747 certificate_manager.go:356] kubernetes.io/kube-apiserver-client-kubelet: Waiting 40h9m29.993063956s for next certificate rotation
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.891340   32747 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.891496   32747 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.891549   32747 manager.go:162] cAdvisor running in container: "/system.slice/kubelet.service"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.892423   32747 fs.go:133] Filesystem UUIDs: map[1cc66862-2b27-4886-916b-25070c61444c:/dev/sda4 5b913b0f-1df1-41f3-8c47-8a5f1f45df8e:/dev/sda3 A7A8-5210:/dev/sda2]
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.892448   32747 fs.go:134] Filesystem partitions: map[/dev/sda3:{mountpoint:/boot major:8 minor:3 fsType:ext4 blockSize:0} /dev/sda4:{mountpoint:/var major:8 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:23 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:25 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:60 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:45 fsType:tmpfs blockSize:0}]
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.896734   32747 manager.go:210] Machine: {Timestamp:2023-11-13 10:41:15.896288437 +0000 UTC m=+0.342921574 CPUVendorID:GenuineIntel NumCores:4 NumPhysicalCores:2 NumSockets:1 CpuFrequency:2800250 MemoryCapacity:16752070656 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:8959cd3fd267a553d1a7d2182831bccf SystemUUID:8959cd3f-d267-a553-d1a7-d2182831bccf BootID:5c7fbc64-84bc-44ae-83c7-50268d4ecf8d Filesystems:[{Device:/dev/sda3 DeviceMajor:8 DeviceMinor:3 Capacity:379452416 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/user/1000 DeviceMajor:0 DeviceMinor:60 Capacity:1675206656 Type:vfs Inodes:408986 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:23 Capacity:8376033280 Type:vfs Inodes:2044930 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:25 Capacity:3350417408 Type:vfs Inodes:819200 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:26 Capacity:4194304 Type:vfs Inodes:1024 HasInodes:true} {Device:/dev/sda4 DeviceMajor:8 DeviceMinor:4 Capacity:160512847872 Type:vfs Inodes:78380480 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:45 Capacity:8376037376 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:161061273600 Scheduler:bfq}] NetworkDevices:[{Name:br-ex MacAddress:42:01:0a:45:12:2f Speed:0 Mtu:1460} {Name:br-int MacAddress:1a:2f:f1:e2:90:fe Speed:0 Mtu:1360} {Name:ens4 MacAddress:42:01:0a:45:12:2f Speed:-1 Mtu:1460} {Name:genev_sys_6081 MacAddress:e2:84:e3:70:97:bb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:66:9b:38:87:c6:c9 Speed:0 Mtu:1360} {Name:ovs-system MacAddress:56:b2:42:32:3e:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:16752070656 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 2] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0} {Id:1 Threads:[1 3] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:GCE InstanceType:n2-standard-4 InstanceID:3277213685289579806}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.896876   32747 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.896925   32747 manager.go:226] Version: {KernelVersion:6.5.4-300.fc39.x86_64 ContainerOsVersion:Fedora CoreOS 38.20231027.3.2 DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897511   32747 container_manager_linux.go:268] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897613   32747 container_manager_linux.go:273] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName:/system.slice/crio.service SystemCgroupsName:/system.slice KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[cpu:{i:{value:500 scale:-3} d:{Dec:<nil>} s:500m Format:DecimalSI} ephemeral-storage:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] CPUManagerPolicy:none CPUManagerPolicyOptions:map[] TopologyManagerScope:container CPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] PodPidsLimit:4096 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms TopologyManagerPolicy:none ExperimentalTopologyManagerPolicyOptions:map[]}
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897631   32747 topology_manager.go:136] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897646   32747 container_manager_linux.go:304] "Creating device plugin manager"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897662   32747 manager.go:135] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897682   32747 server.go:66] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897712   32747 state_mem.go:36] "Initialized new in-memory state store"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.897789   32747 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/crio/crio.sock" URL="unix:///var/run/crio/crio.sock"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.900138   32747 remote_runtime.go:126] "Validated CRI v1 runtime API"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.900190   32747 util_unix.go:103] "Using this endpoint is deprecated, please consider using full URL format" endpoint="/var/run/crio/crio.sock" URL="unix:///var/run/crio/crio.sock"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901243   32747 remote_image.go:98] "Validated CRI v1 image API"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901268   32747 server.go:1011] "Cloud provider determined current node" nodeName="okd-ci-ljhg6-master-a.domain.org"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901280   32747 server.go:1139] "Using root directory" path="/var/lib/kubelet"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901338   32747 kubelet.go:414] "Attempting to sync node with API server"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901354   32747 kubelet.go:307] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901371   32747 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901386   32747 kubelet.go:318] "Adding apiserver pod source"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.901398   32747 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.902300   32747 kuberuntime_manager.go:257] "Container runtime initialized" containerRuntime="cri-o" version="1.28.2" apiVersion="v1"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.902880   32747 reflector.go:533] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://api-int.okd-ci.domain.org:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.902935   32747 reflector.go:148] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://api-int.okd-ci.domain.org:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.903013   32747 reflector.go:533] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://api-int.okd-ci.domain.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dokd-ci-ljhg6-master-a.domain.org&limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.903306   32747 reflector.go:148] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://api-int.okd-ci.domain.org:6443/api/v1/nodes?fieldSelector=metadata.name%3Dokd-ci-ljhg6-master-a.domain.org&limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908572   32747 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-server-current.pem".
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908927   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/rbd"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908947   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/gce-pd"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908957   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/azure-file"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908967   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/vsphere-volume"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.908987   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909003   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909013   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909023   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909040   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909051   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909066   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909076   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/cephfs"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909087   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909104   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909115   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909127   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909137   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909171   32747 plugins.go:639] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.909346   32747 server.go:1174] "Started kubelet"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: Started kubelet.service - Kubernetes Kubelet.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.909632   32747 kubelet.go:1476] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kernel: audit: type=1130 audit(1699872075.908:1413): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.910740   32747 certificate_manager.go:356] kubernetes.io/kubelet-serving: Certificate rotation is enabled
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.910782   32747 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.911197   32747 csi_plugin.go:913] Failed to contact API server when waiting for CSINode publishing: Get "https://api-int.okd-ci.domain.org:6443/apis/storage.k8s.io/v1/csinodes/okd-ci-ljhg6-master-a.domain.org": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.911302   32747 server.go:162] "Starting to listen" address="0.0.0.0" port=10250
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.911761   32747 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"okd-ci-ljhg6-master-a.domain.org.179728a5902e3745", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"okd-ci-ljhg6-master-a.domain.org", UID:"okd-ci-ljhg6-master-a.domain.org", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"okd-ci-ljhg6-master-a.domain.org"}, FirstTimestamp:time.Date(2023, time.November, 13, 10, 41, 15, 909314373, time.Local), LastTimestamp:time.Date(2023, time.November, 13, 10, 41, 15, 909314373, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"kubelet", ReportingInstance:"okd-ci-ljhg6-master-a.domain.org"}': 'Post "https://api-int.okd-ci.domain.org:6443/api/v1/namespaces/default/events": dial tcp 10.69.18.200:6443: connect: connection refused'(may retry after sleeping)
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.912168   32747 server.go:461] "Adding debug handlers to kubelet server"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.912934   32747 ratelimit.go:65] "Setting rate limiting for podresources endpoint" qps=100 burstTokens=10
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.913279   32747 certificate_manager.go:356] kubernetes.io/kubelet-serving: Certificate expiration is 2023-11-17 22:16:14 +0000 UTC, rotation deadline is 2023-11-14 02:00:28.116338156 +0000 UTC
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.913317   32747 volume_manager.go:288] "The desired_state_of_world populator starts"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.913327   32747 volume_manager.go:290] "Starting Kubelet Volume Manager"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.913332   32747 certificate_manager.go:356] kubernetes.io/kubelet-serving: Waiting 15h19m12.203026658s for next certificate rotation
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.913368   32747 kubelet_node_status.go:485] "Error getting the current node from lister" err="node \"okd-ci-ljhg6-master-a.domain.org\" not found"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.913403   32747 oomparser.go:173] error reading /dev/kmsg: read /dev/kmsg: broken pipe
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.913419   32747 oomparser.go:149] exiting analyzeLines. OOM events will not be reported.
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.913433   32747 desired_state_of_world_populator.go:151] "Desired state populator starts to run"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org crio[1872]: time="2023-11-13 10:41:15.913320398Z" level=info msg="Checking image status: quay.io/openshift/okd-content@sha256:0f000acd058731898c827375ac88a228d2b1acf9bb17e3db408b8c3922eb9f58" id=5f2a85d5-1108-44bc-9860-e9d01c2d0b56 name=/runtime.v1.ImageService/ImageStatus
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org crio[1872]: time="2023-11-13 10:41:15.913699371Z" level=info msg="Image quay.io/openshift/okd-content@sha256:0f000acd058731898c827375ac88a228d2b1acf9bb17e3db408b8c3922eb9f58 not found" id=5f2a85d5-1108-44bc-9860-e9d01c2d0b56 name=/runtime.v1.ImageService/ImageStatus
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.914967   32747 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://api-int.okd-ci.domain.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/okd-ci-ljhg6-master-a.domain.org?timeout=10s\": dial tcp 10.69.18.200:6443: connect: connection refused" interval="200ms"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.915022   32747 reflector.go:533] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: Get "https://api-int.okd-ci.domain.org:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.915181   32747 reflector.go:148] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://api-int.okd-ci.domain.org:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.915197   32747 factory.go:153] Registering CRI-O factory
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.915251   32747 factory.go:55] Registering systemd factory
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.915400   32747 factory.go:103] Registering Raw factory
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.915537   32747 manager.go:1186] Started watching for new ooms in manager
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.915945   32747 manager.go:299] Starting recovery of all containers
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.923894   32747 kubelet_network_linux.go:63] "Initialized iptables rules." protocol=IPv4
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.924757   32747 manager.go:304] Recovery completed
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.925629   32747 kubelet_network_linux.go:63] "Initialized iptables rules." protocol=IPv6
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.925650   32747 status_manager.go:216] "Starting to sync pod status with apiserver"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.925669   32747 kubelet.go:2339] "Starting kubelet main sync loop"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.925717   32747 kubelet.go:2363] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.929641   32747 container.go:586] Failed to update stats for container "/": openat2 /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.929984   32747 container.go:586] Failed to update stats for container "/system.slice/system-getty.slice/[email protected]": openat2 /sys/fs/cgroup/memory/system.slice/system-getty.slice/[email protected]/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.930308   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-journald.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-journald.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.930681   32747 reflector.go:533] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.RuntimeClass: Get "https://api-int.okd-ci.domain.org:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.930745   32747 reflector.go:148] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://api-int.okd-ci.domain.org:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 10.69.18.200:6443: connect: connection refused
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.930710   32747 container.go:586] Failed to update stats for container "/system.slice/openshift-gcp-routes.service": openat2 /sys/fs/cgroup/memory/system.slice/openshift-gcp-routes.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931057   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-logind.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-logind.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931058   32747 container.go:586] Failed to update stats for container "/system.slice/chronyd.service": openat2 /sys/fs/cgroup/memory/system.slice/chronyd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931282   32747 container.go:586] Failed to update stats for container "/system.slice/ovsdb-server.service": openat2 /sys/fs/cgroup/memory/system.slice/ovsdb-server.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931413   32747 container.go:586] Failed to update stats for container "/system.slice/rpcbind.service": openat2 /sys/fs/cgroup/memory/system.slice/rpcbind.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931553   32747 container.go:586] Failed to update stats for container "/system.slice/system-sshd\x2dkeygen.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-sshd\x2dkeygen.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931791   32747 container.go:586] Failed to update stats for container "/system.slice/system-systemd\x2dfsck.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-systemd\x2dfsck.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.931811   32747 container.go:586] Failed to update stats for container "/system.slice/system-afterburn\x2dsshkeys.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-afterburn\x2dsshkeys.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932077   32747 container.go:586] Failed to update stats for container "/system.slice/system-serial\x2dgetty.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-serial\x2dgetty.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932124   32747 container.go:586] Failed to update stats for container "/system.slice/crio.service": openat2 /sys/fs/cgroup/memory/system.slice/crio.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932483   32747 container.go:586] Failed to update stats for container "/system.slice/rpc-statd.service": openat2 /sys/fs/cgroup/memory/system.slice/rpc-statd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932588   32747 container.go:586] Failed to update stats for container "/system.slice/system-serial\x2dgetty.slice/[email protected]": openat2 /sys/fs/cgroup/memory/system.slice/system-serial\x2dgetty.slice/[email protected]/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932847   32747 container.go:586] Failed to update stats for container "/system.slice/system-getty.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-getty.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.932975   32747 container.go:586] Failed to update stats for container "/system.slice/sshd.service": openat2 /sys/fs/cgroup/memory/system.slice/sshd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.933177   32747 container.go:586] Failed to update stats for container "/system.slice": openat2 /sys/fs/cgroup/memory/system.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.933391   32747 container.go:586] Failed to update stats for container "/system.slice/ovs-vswitchd.service": openat2 /sys/fs/cgroup/memory/system.slice/ovs-vswitchd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.933542   32747 container.go:586] Failed to update stats for container "/system.slice/NetworkManager.service": openat2 /sys/fs/cgroup/memory/system.slice/NetworkManager.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.933833   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-resolved.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-resolved.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.933866   32747 container.go:586] Failed to update stats for container "/system.slice/dbus-broker.service": openat2 /sys/fs/cgroup/memory/system.slice/dbus-broker.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934217   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-homed.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-homed.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934217   32747 container.go:586] Failed to update stats for container "/system.slice/irqbalance.service": openat2 /sys/fs/cgroup/memory/system.slice/irqbalance.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934544   32747 container.go:586] Failed to update stats for container "/system.slice/system-modprobe.slice": openat2 /sys/fs/cgroup/memory/system.slice/system-modprobe.slice/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934673   32747 container.go:586] Failed to update stats for container "/system.slice/kubelet.service": openat2 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934758   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-udevd.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-udevd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: W1113 10:41:15.934989   32747 container.go:586] Failed to update stats for container "/system.slice/systemd-userdbd.service": openat2 /sys/fs/cgroup/memory/system.slice/systemd-userdbd.service/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.947909   32747 kubelet_node_status.go:376] "Setting node annotation to enable volume controller attach/detach"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949541   32747 kubelet_node_status.go:424] "Adding label from cloud provider" labelKey="beta.kubernetes.io/instance-type" labelValue="n2-standard-4"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949560   32747 kubelet_node_status.go:426] "Adding node label from cloud provider" labelKey="node.kubernetes.io/instance-type" labelValue="n2-standard-4"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949570   32747 kubelet_node_status.go:437] "Adding node label from cloud provider" labelKey="failure-domain.beta.kubernetes.io/zone" labelValue="europe-west6-a"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949577   32747 kubelet_node_status.go:439] "Adding node label from cloud provider" labelKey="topology.kubernetes.io/zone" labelValue="europe-west6-a"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949585   32747 kubelet_node_status.go:443] "Adding node label from cloud provider" labelKey="failure-domain.beta.kubernetes.io/region" labelValue="europe-west6"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.949593   32747 kubelet_node_status.go:445] "Adding node label from cloud provider" labelKey="topology.kubernetes.io/region" labelValue="europe-west6"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.949624   32747 container_manager_linux.go:883] "Unable to get rootfs data from cAdvisor interface" err="unable to find data in memory cache"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.950875   32747 kubelet_node_status.go:696] "Recording event message for node" node="okd-ci-ljhg6-master-a.domain.org" event="NodeHasSufficientMemory"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.950907   32747 kubelet_node_status.go:696] "Recording event message for node" node="okd-ci-ljhg6-master-a.domain.org" event="NodeHasNoDiskPressure"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.950921   32747 kubelet_node_status.go:696] "Recording event message for node" node="okd-ci-ljhg6-master-a.domain.org" event="NodeHasSufficientPID"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.951524   32747 cpu_manager.go:215] "Starting CPU manager" policy="none"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.951544   32747 cpu_manager.go:216] "Reconciling" reconcilePeriod="10s"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.951560   32747 state_mem.go:36] "Initialized new in-memory state store"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.953681   32747 policy_none.go:49] "None policy: Start"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.954300   32747 memory_manager.go:169] "Starting memorymanager" policy="None"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: I1113 10:41:15.954328   32747 state_mem.go:35] "Initializing new in-memory state store"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org kubenswrapper[32747]: E1113 10:41:15.956077   32747 kubelet.go:1556] "Failed to start ContainerManager" err="failed to get rootfs info: unable to find data in memory cache"
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 10:41:15 okd-ci-ljhg6-master-a.domain.org systemd[1]: kubelet.service: Failed with result 'exit-code'.

vrutkovs · 2023-11-13T11:39:31Z

vrutkovs
Nov 13, 2023
Maintainer

Please attach (or upload to the public file sharing service) must-gather archive

12 replies

llomgui Nov 13, 2023
Author

@vrutkovs About cri-o version, do you want me to specify 1.29?

vrutkovs Nov 13, 2023
Maintainer

No, it should be 1.27, matching the kubelet. Not sure why 1.28 or 1.29 leaked in

llomgui Nov 13, 2023
Author

From okd-copr repo. I removed it.

madpearl Nov 13, 2023

Yes, the cri-o version should be specified, because during the upgrade to v4.14 of the OKD it was a message, that lower version of cri-o for kubelet will not run anymore.

llomgui Nov 13, 2023
Author

I will switch to this new version with cri-o 1.27 on the master node. I will let you know.

vrutkovs · 2023-11-13T18:01:42Z

vrutkovs
Nov 13, 2023
Maintainer

Lets track this in #1798

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OS update error during latest update 4.14.0-0.okd-2023-11-12-042703 #1794

{{title}}

Replies: 2 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

OS update error during latest update 4.14.0-0.okd-2023-11-12-042703 #1794

llomgui Nov 13, 2023

Replies: 2 comments · 12 replies

vrutkovs Nov 13, 2023 Maintainer

llomgui Nov 13, 2023 Author

vrutkovs Nov 13, 2023 Maintainer

llomgui Nov 13, 2023 Author

madpearl Nov 13, 2023

llomgui Nov 13, 2023 Author

vrutkovs Nov 13, 2023 Maintainer

llomgui
Nov 13, 2023

Replies: 2 comments 12 replies

vrutkovs
Nov 13, 2023
Maintainer

llomgui Nov 13, 2023
Author

vrutkovs Nov 13, 2023
Maintainer

llomgui Nov 13, 2023
Author

llomgui Nov 13, 2023
Author

vrutkovs
Nov 13, 2023
Maintainer