Integration with K8s, "Modify Kubelet config" instructions do not work.

[adamwoolhether]

I've managed to solve the issue, but I'm not sure if the manual config was needed due to something on my end or if it's an issue with documentation.

What are the steps to reproduce this issue?

1. $ cat > /etc/default/kubelet <<EOF
   KUBELET_EXTRA_ARGS=--container-runtime=remote
   --container-runtime-endpoint=unix:///var/run/singularity.sock
   --image-service-endpoint=unix:///var/run/singularity.sock
   EOF
2. sudo systemctl restart kubelet

What happens?

K8's does not use the singular runtime interface.

What were you expecting to happen?

When applying the arch-extender pod, K8's should use the new runtime interface to pull the images from Image: cloud.sylabs.io/library/slurm/arch-scheduler. However this results in ImagePullBackOff.

Instead I had to edit /etc/kubernetes/kubelet.env and add the KUBELET_EXTRA_ARGS var to /etc/systemd/system/kubelet.service

Any logs, error output, comments, etc?

All pods container to run with default cri, as determined with crictl
crictl pods
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.

k logs sif-scheduler-extender
Error from server (BadRequest): container "sif-scheduler-extender" in pod "sif-scheduler-extender" is waiting to start: trying and failing to pull image
...

Environment?

OS distribution and version: CentOS Linux release 7.9.2009 (Core)

go version: go1.11 linux/amd64

go env: GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/arch-scheduler/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build165916451=/tmp/go-build -gno-record-gcc-switches

Singularity-CRI version: 1.0.0-beta.5

Singularity version: 3.6.4-1.el7

Kubernetes version: 1.17.4