Cilium CNi with k8s does not work with SELinux in permissive mode

[alirezaghey]

Description

Cilium CNi with k8s does not work with SELinux in permissive mode.

Impact

You need to disable SELinux for Cilium to work.

Environment and steps to reproduce

1. Set-up: Latest Flatcar Linux, with k8s, and latest Cilium CNi plugin.
2. Task: Trying to boot the CNi plugin

Expected behavior

Expected Cilium to run with SELinux in permissive mode.

Additional information

Disabling SELinux does the trick. Probably, the necessary policies and labels are not in place. Related to #673

[tormath1]

Hello @alirezaghey, thanks for the report.

Can you share some error logs and the Cilium version? Flatcar is currently tested with Cilium in permissive mode. I notified some issues on the latest 1.13.0-rc* but nothing related to SELinux.

[tormath1]

Hello again @alirezaghey,

I just got an illumination. We indeed faced this issue, and it's currently patched in this commit: flatcar/mantle@e8e9751

This is required even with Permissive mode. Can be dropped once spc_t
is supported on Flatcar.

[alirezaghey]

Hi @tormath1

Sorry for the incomplete issue.

$ cat /etc/os-release 
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3227.2.4
VERSION_ID=3227.2.4
BUILD_ID=2022-10-27-1321
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3227.2.4 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3227.2.4:*:*:*:*:*:*:*"

$ cilium version
cilium-cli: v0.12.4 compiled with go1.19.1 on linux/amd64
cilium image (default): v1.12.2
cilium image (stable): v1.12.3
cilium image (running): v1.12.2

Part of journalctl -u kubelet output. I think the invalid argument: unknown errors had with SELinux to do:

...
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: E1103 10:57:39.287131   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.310984   15588 topology_manager.go:205] "Topology Admit Handler"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363545   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bpf-maps\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-bpf-maps\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363622   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-cgroup\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-cilium-cgroup\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363695   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-proc-sys-net\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-host-proc-sys-net\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363726   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-path\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-cni-path\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363775   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-xtables-lock\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363805   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"clustermesh-secrets\" (UniqueName: \"kubernetes.io/secret/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-clustermesh-secrets\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363848   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-config-path\" (UniqueName: \"kubernetes.io/configmap/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-cilium-config-path\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363901   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hubble-tls\" (UniqueName: \"kubernetes.io/projected/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-hubble-tls\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363932   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-etc-cni-netd\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.363976   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rxc\" (UniqueName: \"kubernetes.io/projected/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-kube-api-access-59rxc\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.364002   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-run\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-cilium-run\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.364043   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostproc\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-hostproc\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.364070   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-proc-sys-kernel\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-host-proc-sys-kernel\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.364098   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fffeec14-b0c2-424e-a8e6-0f1ab5924c32-lib-modules\") pod \"cilium-4dctx\" (UID: \"fffeec14-b0c2-424e-a8e6-0f1ab5924c32\") " pod="kube-system/cilium-4dctx"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.623612   15588 topology_manager.go:205] "Topology Admit Handler"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.665804   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdxk\" (UniqueName: \"kubernetes.io/projected/ab0dc6c1-393c-4882-a555-b412b1d4ca90-kube-api-access-frdxk\") pod \"cilium-operator-bc4d5b54-tlll9\" (UID: \"ab0dc6c1-393c-4882-a555-b412b1d4ca90\") " pod="kube-system/cilium-operator-bc4d5b54-tlll9"
Nov 03 10:57:39 az-k8s-master1.novalocal kubelet[15588]: I1103 10:57:39.666386   15588 reconciler.go:357] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cilium-config-path\" (UniqueName: \"kubernetes.io/configmap/ab0dc6c1-393c-4882-a555-b412b1d4ca90-cilium-config-path\") pod \"cilium-operator-bc4d5b54-tlll9\" (UID: \"ab0dc6c1-393c-4882-a555-b412b1d4ca90\") " pod="kube-system/cilium-operator-bc4d5b54-tlll9"
Nov 03 10:57:44 az-k8s-master1.novalocal kubelet[15588]: E1103 10:57:44.298741   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:57:49 az-k8s-master1.novalocal kubelet[15588]: E1103 10:57:49.300708   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:57:54 az-k8s-master1.novalocal kubelet[15588]: E1103 10:57:54.302119   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:57:59 az-k8s-master1.novalocal kubelet[15588]: E1103 10:57:59.304829   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:04 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:04.306102   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:08.766804   15588 remote_runtime.go:474] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" containerID="b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f"
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:08.767075   15588 kuberuntime_manager.go:862] init container &Container{Name:mount-cgroup,Image:quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36,Command:[sh -ec cp /usr/bin/cilium-mount /hostbin/cilium-mount;
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-mount" $CGROUP_ROOT;
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: rm /hostbin/cilium-mount
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: ],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:CGROUP_ROOT,Value:/run/cilium/cgroupv2,ValueFrom:nil,},EnvVar{Name:BIN_PATH,Value:/opt/cni/bin,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:hostproc,ReadOnly:false,MountPath:/hostproc,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-path,ReadOnly:false,MountPath:/hostbin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-59rxc,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[SYS_ADMIN SYS_CHROOT SYS_PTRACE],Drop:[ALL],},Privileged:nil,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:spc_t,Level:s0,},RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod cilium-4dctx_kube-system(fffeec14-b0c2-424e-a8e6-0f1ab5924c32): RunContainerError: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown
Nov 03 10:58:08 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:08.767242   15588 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mount-cgroup\" with RunContainerError: \"failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown\"" pod="kube-system/cilium-4dctx" podUID=fffeec14-b0c2-424e-a8e6-0f1ab5924c32
Nov 03 10:58:09 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:09.307343   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: W1103 10:58:11.380170   15588 manager.go:1174] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffeec14_b0c2_424e_a8e6_0f1ab5924c32.slice/cri-containerd-b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f.scope WatchSource:0}: task b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f not found: not found
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.705806   15588 remote_runtime.go:474] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" containerID="e71373372c645ba281a4a1fc5b820a19953f257de59ca8d1b53a4a4a75b3e0e1"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.706747   15588 kuberuntime_manager.go:862] init container &Container{Name:mount-cgroup,Image:quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36,Command:[sh -ec cp /usr/bin/cilium-mount /hostbin/cilium-mount;
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-mount" $CGROUP_ROOT;
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: rm /hostbin/cilium-mount
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: ],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:CGROUP_ROOT,Value:/run/cilium/cgroupv2,ValueFrom:nil,},EnvVar{Name:BIN_PATH,Value:/opt/cni/bin,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:hostproc,ReadOnly:false,MountPath:/hostproc,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-path,ReadOnly:false,MountPath:/hostbin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-59rxc,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[SYS_ADMIN SYS_CHROOT SYS_PTRACE],Drop:[ALL],},Privileged:nil,SELinuxOptions:&SELinuxOptions{User:,Role:,Type:spc_t,Level:s0,},RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod cilium-4dctx_kube-system(fffeec14-b0c2-424e-a8e6-0f1ab5924c32): RunContainerError: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.706838   15588 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mount-cgroup\" with RunContainerError: \"failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown\"" pod="kube-system/cilium-4dctx" podUID=fffeec14-b0c2-424e-a8e6-0f1ab5924c32
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: I1103 10:58:11.793081   15588 scope.go:115] "RemoveContainer" containerID="b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: I1103 10:58:11.794325   15588 scope.go:115] "RemoveContainer" containerID="b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.797865   15588 remote_runtime.go:531] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f\": container is already in removing state" containerID="b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.797946   15588 kuberuntime_container.go:777] failed to remove pod init container "mount-cgroup": rpc error: code = Unknown desc = failed to set removing state for container "b9de8946329cacabab9d6a1ed07cb6b33ad0a6b1c7aecf026712ae81d2f3553f": container is already in removing state; Skipping pod "cilium-4dctx_kube-system(fffeec14-b0c2-424e-a8e6-0f1ab5924c32)"
Nov 03 10:58:11 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:11.798419   15588 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mount-cgroup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=mount-cgroup pod=cilium-4dctx_kube-system(fffeec14-b0c2-424e-a8e6-0f1ab5924c32)\"" pod="kube-system/cilium-4dctx" podUID=fffeec14-b0c2-424e-a8e6-0f1ab5924c32
Nov 03 10:58:12 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:12.799302   15588 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mount-cgroup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=mount-cgroup pod=cilium-4dctx_kube-system(fffeec14-b0c2-424e-a8e6-0f1ab5924c32)\"" pod="kube-system/cilium-4dctx" podUID=fffeec14-b0c2-424e-a8e6-0f1ab5924c32
Nov 03 10:58:14 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:14.308851   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:14 az-k8s-master1.novalocal kubelet[15588]: W1103 10:58:14.672607   15588 manager.go:1174] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffeec14_b0c2_424e_a8e6_0f1ab5924c32.slice/cri-containerd-e71373372c645ba281a4a1fc5b820a19953f257de59ca8d1b53a4a4a75b3e0e1.scope WatchSource:0}: task e71373372c645ba281a4a1fc5b820a19953f257de59ca8d1b53a4a4a75b3e0e1 not found: not found
Nov 03 10:58:19 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:19.311049   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:24 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:24.313193   15588 kubelet.go:2373] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Nov 03 10:58:29 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:29.098670   15588 remote_runtime.go:474] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" containerID="2a687445463a9dc2e66a1a5e3361e9a4380e6b32d2fe067a91d2c3f288230212"
Nov 03 10:58:29 az-k8s-master1.novalocal kubelet[15588]: E1103 10:58:29.098828   15588 kuberuntime_manager.go:862] init container &Container{Name:mount-cgroup,Image:quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36,Command:[sh -ec cp /usr/bin/cilium-mount /hostbin/cilium-mount;
...

Let me know, if I can provide you with other information. I'm a novice with K8s and Flatcar, so sorry if the information isn't as expected.

I just saw the update you posted, but posting this for what it's worth :).

[zach-source]

In case you run into this problem. I was able to work around this with just replacing spc_t with unconfined_t. The simplest method of doing this is doing a post render with the helm chart, but you can also modify the chart by hand. If you're deploying yaml directly, just pipe the yaml through sed.
Example:

helm install ... cilium --postrender 'sed "s/spc_t/unconfined_t/g"'

[tormath1]

Fixed in: flatcar/scripts#917

[ryan0x44]

Thanks for fixing this! Just FYI, this issue is still linked to as a 'known issue' on https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-kubernetes/

[tormath1]

@ryan0x44 thanks for the heads-up - the issue is still referenced because it's not yet landed on Stable (it should arrive with the next release).