Localfile is trying to get image from an endpoint

[sanketmlad]

What are the steps to reproduce this issue?

1. Install Kubernetes with kubeadm

2. Install singularity-cri from master branch

3. modify kubeadm-flags to point singularity socket
   sudo cat /var/lib/kubelet/kubeadm-flags.env
   #KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf --container-runtime-endpoint=unix:///var/run/singularity.sock --image-service-endpoint=unix:///var/run/singularity.sock"
   KUBELET_KUBEADM_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///var/run/singularity.sock --image-service-endpoint=unix:///var/run/singularity.sock

4. Run sycri and kubelet

5. Spin up hello world using following yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   name: hello-world-singularity
   spec:
   replicas: 1
   selector:
   matchLabels:
   app: hello-world-singularity
   template:
   metadata:
   labels:
   app: hello-world-singularity
   spec:
   containers:
   - name: hello-world-singularity
   image: local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif
   ports:
   - containerPort: 8080

6. Describe pot shows following error
   Error: ImagePullBackOff
   Pulling image "local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif"
   Failed to pull image "local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif": rpc error: code = Unknown desc = Error response from daemon: Get https://local.file/v2/: dial tcp: lookup local.file: no such host

What happens?
When I spin up a container using local file, it tries to reach to some hostname local.file and gives message "local.file: no such host"

What were you expecting to happen?
It should pick up from local file

Any logs, error output, comments, etc?
Failed to pull image "local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif": rpc error: code = Unknown desc = Error response from daemon: Get https://local.file/v2/: dial tcp: lookup local.file: no such host

Environment?

OS distribution and version:: Ubuntu 18.04 Kubernetes master and slave on Virtual box

go version: go1.12.7 linux/amd64

go env: …

Singularity-CRI version: 1.0.0-beta.4.7.gf95f62d.dirty

Singularity version: 3.3.0-rc.1.455.gd8c768aac

Kubernetes version: v1.15.1

[sashayakovtseva]

@sanketmlad Can you do kubectl get po -o wide and
then kubectl describe no <pod's-node>.

The error looks like pod was allocated on non-singularity node...

[sashayakovtseva]

We don't have deamon :)

[sanketmlad]

@sashayakovtseva that output is posted in the bug.
But you are right, it might have picked non-singularity node.
How to correct that?

[sashayakovtseva]

For each singularity node do:
kubectl label no <node-name> runtime=singularity.

Use nodeSelector on pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-singularity
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-singularity
  template:
    metadata:
      labels:
        app: hello-world-singularity
    spec:
      nodeSelector:
        runtime: singularity
      containers:
      - name: hello-world-singularity
        image: local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif
        ports:
        - containerPort: 8080

[sanketmlad]

Hi Sasha
I tried your suggestion

Below is the output for describe command

Name: hello-world-singularity-77f6d7bc46-pptnj
Namespace: default
Priority: 0
Node: kube-singularity-slave/100.106.3.124
Start Time: Fri, 26 Jul 2019 02:18:41 -0500
Labels: app=hello-world-singularity
pod-template-hash=77f6d7bc46
Annotations:
Status: Pending
IP: 10.40.0.3
Controlled By: ReplicaSet/hello-world-singularity-77f6d7bc46
Containers:
hello-world-singularity:
Container ID:
Image: local.file/home/sanket/hello-world-singularity/hello-world-singularity.sif
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wws94 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-wws94:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wws94
Optional: false
QoS Class: BestEffort
Node-Selectors: runtime=singularity
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

---

Normal Pulling 6d8h (x2 over 6d8h) kubelet, kube-singularity-slave Pulling image "local.file/home/elansad/hello-world-singularity/hello-world-singularity.sif"
Warning Failed 6d8h (x2 over 6d8h) kubelet, kube-singularity-slave Failed to pull image "local.file/home/elansad/hello-world-singularity/hello-world-singularity.sif": rpc error: code = Unknown desc = Error response from daemon: Get https://local.file/v2/: dial tcp: lookup local.file: no such host
Warning Failed 6d8h (x2 over 6d8h) kubelet, kube-singularity-slave Error: ErrImagePull
Normal BackOff 6d8h (x3 over 6d8h) kubelet, kube-singularity-slave Back-off pulling image "local.file/home/elansad/hello-world-singularity/hello-world-singularity.sif"
Warning Failed 6d8h (x3 over 6d8h) kubelet, kube-singularity-slave Error: ImagePullBackOff
Normal Scheduled 31s default-scheduler Successfully assigned default/hello-world-singularity-77f6d7bc46-pptnj to kube-singularity-slave

[sanketmlad]

BTW I have not installed singularity on Kubernetes slave machine
Should I install and try again?

[sashayakovtseva]

Yes, of course :)

[sanketmlad]

That did not affect the error. Getting same error after installing same version of singularity on slave node of kubernetes

[sanketmlad]

I suspect that the prefix local.file is still not being considered, and it is trying to get to web.
"Error response from daemon: Get https://local.file/v2/: dial tcp: lookup local.file: no such host"
It is assuming local.file is an endpoint

[sashayakovtseva]

@sanketmlad Still the error output looks like it is docker, not Singularity.
On your node where you want to run SIF, do:

1. Install Singularity
2. Install and launch Singularity-CRI (refer to docs)
3. Label node with runtime=singularity (as described above), make sure that label doesn't appear on other nodes ( that where not set up with steps 1-2 )
4. Launch pod with kubectl apply -f pod.yaml

LMK if that works.

[sashayakovtseva]

Hello @sanketmlad,
Did you manage to make it work so far?