sycri can't create container

[malixian]

my kubelet.service config is:

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kube/bin/kubelet
--address=10.2.152.182
--allow-privileged=true
--anonymous-auth=false
--authentication-token-webhook
--authorization-mode=Webhook
--client-ca-file=/etc/kubernetes/ssl/ca.pem
--cluster-dns=10.70.0.2
--cluster-domain=cluster.local.
--cni-bin-dir=/opt/kube/bin
--cni-conf-dir=/etc/cni/net.d
--fail-swap-on=false
--hairpin-mode hairpin-veth
--hostname-override=10.2.152.182
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig
--max-pods=110
--network-plugin=cni
--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1
--register-node=true
--root-dir=/var/lib/kubelet
--tls-cert-file=/etc/kubernetes/ssl/kubelet.pem
--tls-private-key-file=/etc/kubernetes/ssl/kubelet-key.pem
--v=2
--container-runtime=remote
--container-runtime-endpoint=unix:///var/run/singularity.sock
--image-service-endpoint=unix:///var/run/singularity.sock

ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

i think this config is ok, but pod events have some error:

Error: could not create container: could not spawn container: could not create oci bundle: could not create SIF bundle: failed to load SIF image /var/lib/singularity/cf5d9eea227371037e614fc7dec7c1f437a6398f9b08250b89ef5c92aab7e737: image format not recognized

[sashayakovtseva]

Hi @malixian,

Please provide the following information:

• OS distribution and version
• go version
• Singularity-CRI version (sycri version)
• singularity version
• kubectl version
• pod specification you are trying to run

BTW do you have docker running? I see you require docker.service in kubelet service, which is not needed if you use other runtime.

[malixian]

thanks @sashayakovtseva ,
Here is my environment:
1. arch is arm64
2. singularity version 3.4.0-1
3.kubelet version v1.13.5
consider with my arch is arm64, so customize to build a image in this node and upload to sylab.io.
pod yaml is sample:

> apiVersion: v1
> kind: Pod
> metadata:
>   name: test-arm64
>   namespace: default
> spec:
>   containers:
>     name: test-arm64
>     image: cloud.sylabs.io/malixian/default/test-arm64:latest
>   nodeSelector:
>     beta.kubernetes.io/arch: arm64

journalctl -u kubelet -e, find error:

1.provider.go:116] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
2.SyncLoop (PLEG): "test-arm64_default(d938b836-e003-11e9-9f5c-ac1f6bac1d10)", event: &pleg.PodLifecycleEvent{ID:"d938b836-e003-11e9-9f5c-ac1f6bac1d10", Type:"ContainerDied,Data:"29950f6fcfc74d1e6268caba276c898280949621a5e31fc9e9ff91e50b4a360e"}

journalctl -u sycri -e, find error:

Sep 26 14:20:40 localhost sycri[28374]:E0926 14:20:40.218152   28374 main.go:276] /runtime.v1alpha2.ImageService/PullImage
Sep 26 14:20:40 localhost sycri[28374]:         Request: {"image":{"image":"cloud.sylabs.io/malixian/default/test-arm64:latest"}}
Sep 26 14:20:40 localhost sycri[28374]:         Response: null
Sep 26 14:20:40 localhost sycri[28374]:         Error: rpc error: code = Internal desc = could not get cloud.sylabs.io/malixian/default/test-arm64:latest image metadata: could not get library image info: error making request to server:
Sep 26 14:20:40 localhost sycri[28374]:         Get https://library.sylabs.io/v1/images/malixian/default/test-arm64:latest: net/http: TLS handshake timeout
Sep 26 14:34:31 localhost sycri[28374]: Calico CNI releasing IP address
Sep 26 14:34:31 localhost sycri[28374]: Calico CNI deleting device in netns /var/run/singularity/pods/19d9118ee6ac1e19ce475a4650ae0021f752c635009b0888b3f5d03b0a803f2a/namesp
Sep 26 14:34:31 localhost sycri[28374]: Calico CNI deleted device in netns /var/run/singularity/pods/19d9118ee6ac1e19ce475a4650ae0021f752c635009b0888b3f5d03b0a803f2a/namespa
Sep 26 14:34:36 localhost sycri[28374]: Calico CNI IPAM request count IPv4=1 IPv6=0
Sep 26 14:34:36 localhost sycri[28374]: Calico CNI IPAM handle=k8s-pod-network.1a08e33abd1973f28f4b6d0ec580ca335376eaba7e052f29dc94c93a10bb6696
Sep 26 14:34:36 localhost sycri[28374]: Calico CNI IPAM assigned addresses IPv4=[172.22.20.159] IPv6=[]
Sep 26 14:34:36 localhost sycri[28374]: Calico CNI using IPs: [172.22.20.159/32]
Sep 26 14:35:00 localhost sycri[28374]: E0926 14:35:00.032013   28374 container.go:177] Could not fetch container 393b15294a318ec6dd6fcb96a02a663345ec346f937e904982fa6acfd02
Sep 26 14:35:30 localhost sycri[28374]: E0926 14:35:30.725947   28374 pod.go:164] Could not update pod state: could not get pod state: could not query state: FATAL:   no con
Sep 26 14:45:46 localhost sycri[28374]: E0926 14:45:46.791990   28374 container.go:177] Could not fetch container 1e60e5d99769517eaa4add0bc7c70daecd053171434ff4ecaba5dd517f9
Sep 26 14:56:14 localhost sycri[28374]: E0926 14:56:14.904313   28374 container.go:177] Could not fetch container 498ddec28abee523f56c20ab1f173e690e382b4c5d3ec10c1f89993b701

it looks like image not't exist，but i have pushed image with

> singualrity push image.sif library://malixian/default/test-arm64

i try to deploy sycri in x86,but also appear some error：
journalctl -u kubelet -e

pod_workers.go:190] Error syncing pod e0f2d839-e038-11e9-9f5c-ac1f6bac1d10 ("image-service-deployment-655d89d94d-rfl5f_default(e0f2d839-e038-11e9-9f5c-ac1f6bac1d10)"), skipping: failed to "StartContainer" for "image-server" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=image-server pod=image-service-deployment-655d89d94d-rfl5f_default(e0f2d11e9-9f5c-ac1f6bac1d10)"
cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to ge
9月 26 17:06:18 comput1 kubelet[2792]: E0926 17:06:18.623068    2792 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to get device for dir "/run": could not find device with major: 0, minor: 20 in cached partitions map.

journalctl -u sycri -e

9月 26 16:43:55 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:44:41 comput1 sycri[1787]: E0926 16:44:41.549846    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:44:41 comput1 sycri[1787]: Request: {"container_id":"adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ead5"}
9月 26 16:44:41 comput1 sycri[1787]: Response: null
9月 26 16:44:41 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:05 comput1 sycri[1787]: E0926 16:46:05.197723    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:46:05 comput1 sycri[1787]: Request: {"container_id":"d33181a4531a36e3f365c1d9b3b6107137a28c64ad752d762ce603fe1b95a7cd"}
9月 26 16:46:05 comput1 sycri[1787]: Response: null
9月 26 16:46:05 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:06 comput1 sycri[1787]: E0926 16:46:06.262393    1787 container.go:177] Could not fetch container adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ea
9月 26 16:48:50 comput1 sycri[1787]: E0926 16:48:50.075495    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:48:50 comput1 sycri[1787]: Request: {"container_id":"82800d31b039ae364530530a4138e1ac245e222c6f80b83c74c491ea0f84a94e"}
9月 26 16:48:50 comput1 sycri[1787]: Response: null

[sashayakovtseva]

@malixian What version of singularity-cri you are using?

Can you confirm the same network error appears if you do singularity pull library://malixian/default/test-arm64:latest on that host?

[malixian]

@malixian What version of singularity-cri you are using?

Can you confirm the same network error appears if you do singularity pull library://malixian/default/test-arm64:latest on that host?

it's ok in that host.
And i try to deploy sycri in x86,but also appear some error：
journalctl -u kubelet -e

pod_workers.go:190] Error syncing pod e0f2d839-e038-11e9-9f5c-ac1f6bac1d10 ("image-service-deployment-655d89d94d-rfl5f_default(e0f2d839-e038-11e9-9f5c-ac1f6bac1d10)"), skipping: failed to "StartContainer" for "image-server" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=image-server pod=image-service-deployment-655d89d94d-rfl5f_default(e0f2d11e9-9f5c-ac1f6bac1d10)"
cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to ge
9月 26 17:06:18 comput1 kubelet[2792]: E0926 17:06:18.623068    2792 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to get device for dir "/run": could not find device with major: 0, minor: 20 in cached partitions map.

journalctl -u sycri -e

9月 26 16:43:55 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:44:41 comput1 sycri[1787]: E0926 16:44:41.549846    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:44:41 comput1 sycri[1787]: Request: {"container_id":"adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ead5"}
9月 26 16:44:41 comput1 sycri[1787]: Response: null
9月 26 16:44:41 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:05 comput1 sycri[1787]: E0926 16:46:05.197723    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:46:05 comput1 sycri[1787]: Request: {"container_id":"d33181a4531a36e3f365c1d9b3b6107137a28c64ad752d762ce603fe1b95a7cd"}
9月 26 16:46:05 comput1 sycri[1787]: Response: null
9月 26 16:46:05 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:06 comput1 sycri[1787]: E0926 16:46:06.262393    1787 container.go:177] Could not fetch container adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ea
9月 26 16:48:50 comput1 sycri[1787]: E0926 16:48:50.075495    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:48:50 comput1 sycri[1787]: Request: {"container_id":"82800d31b039ae364530530a4138e1ac245e222c6f80b83c74c491ea0f84a94e"}
9月 26 16:48:50 comput1 sycri[1787]: Response: null

[malixian]

@malixian What version of singularity-cri you are using?
Can you confirm the same network error appears if you do singularity pull library://malixian/default/test-arm64:latest on that host?

it's ok in that host.
And i try to deploy sycri in x86,but also appear some error：
yaml file is official example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: image-service-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: image-service
  template:
    metadata:
      labels:
        app: image-service
      name: image-service
      namespace: default
    spec:
      containers:
      - name: image-server
        image: cloud.sylabs.io/sashayakovtseva/test/image-server
        ports:
        - containerPort: 8080
      securityContext:
        runAsUser: 1000
      nodeSelector:
        kubernetes.io/hostname: 10.18.127.1
---
apiVersion: v1
kind: Service
metadata:
  name: image-service
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: image-service

journalctl -u kubelet -e

pod_workers.go:190] Error syncing pod e0f2d839-e038-11e9-9f5c-ac1f6bac1d10 ("image-service-deployment-655d89d94d-rfl5f_default(e0f2d839-e038-11e9-9f5c-ac1f6bac1d10)"), skipping: failed to "StartContainer" for "image-server" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=image-server pod=image-service-deployment-655d89d94d-rfl5f_default(e0f2d11e9-9f5c-ac1f6bac1d10)"
cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to ge
9月 26 17:06:18 comput1 kubelet[2792]: E0926 17:06:18.623068    2792 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/run": failed to get device for dir "/run": could not find device with major: 0, minor: 20 in cached partitions map.

journalctl -u sycri -e

9月 26 16:43:55 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:44:41 comput1 sycri[1787]: E0926 16:44:41.549846    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:44:41 comput1 sycri[1787]: Request: {"container_id":"adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ead5"}
9月 26 16:44:41 comput1 sycri[1787]: Response: null
9月 26 16:44:41 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:05 comput1 sycri[1787]: E0926 16:46:05.197723    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:46:05 comput1 sycri[1787]: Request: {"container_id":"d33181a4531a36e3f365c1d9b3b6107137a28c64ad752d762ce603fe1b95a7cd"}
9月 26 16:46:05 comput1 sycri[1787]: Response: null
9月 26 16:46:05 comput1 sycri[1787]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 26 16:46:06 comput1 sycri[1787]: E0926 16:46:06.262393    1787 container.go:177] Could not fetch container adcec04f0d24bf166b94b202dc7b1ebdd2dae6a0437c6982cc5f8111f8c6ea
9月 26 16:48:50 comput1 sycri[1787]: E0926 16:48:50.075495    1787 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 26 16:48:50 comput1 sycri[1787]: Request: {"container_id":"82800d31b039ae364530530a4138e1ac245e222c6f80b83c74c491ea0f84a94e"}
9月 26 16:48:50 comput1 sycri[1787]: Response: null

[sashayakovtseva]

@malixian Is this is not an issue anymore? Looks like you have accidentally closed it.

And I need a full output of kubectl describe no <your node> and sycri version.

[malixian]

@sashayakovtseva yes,you're right, sycri version is 1.0.0-beta.5. node information is

Name:               10.18.127.3
Roles:              node
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=10.18.127.3
                    kubernetes.io/role=node
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 12 Jun 2019 16:20:55 +0800
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 26 Sep 2019 17:56:07 +0800   Wed, 12 Jun 2019 16:20:55 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 26 Sep 2019 17:56:07 +0800   Wed, 12 Jun 2019 16:20:55 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 26 Sep 2019 17:56:07 +0800   Wed, 12 Jun 2019 16:20:55 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 26 Sep 2019 17:56:07 +0800   Tue, 24 Sep 2019 17:55:59 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  10.18.127.3
  Hostname:    10.18.127.3
Capacity:
 cpu:                32
 ephemeral-storage:  511750Mi
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             65393136Ki
 pods:               110
Allocatable:
 cpu:                32
 ephemeral-storage:  482947890401
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             65290736Ki
 pods:               110
System Info:
 Machine ID:                 2e4e94b510a14392ab58491d3e377c96
 System UUID:                00000000-0000-0000-0000-AC1F6BAC404E
 Boot ID:                    d5e7100c-4a2f-4a96-a776-179cf47676ef
 Kernel Version:             3.10.0-957.21.2.el7.x86_64
 OS Image:                   CentOS Linux 7 (Core)
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.2
 Kubelet Version:            v1.13.5
 Kube-Proxy Version:         v1.13.5
PodCIDR:                     172.22.1.0/24
Non-terminated Pods:         (40 in total)
  Namespace                  Name                                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                                         ------------  ----------  ---------------  -------------  ---
  kube-system                calico-kube-controllers-84db645bdf-ggptb                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kube-system                calico-node-r85wt                                            250m (0%)     0 (0%)      0 (0%)           0 (0%)         106d
  kube-system                coredns-7c5785cbcc-2f4r6                                     100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     2d
  kube-system                coredns-7c5785cbcc-pqvfs                                     100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     2d
  kube-system                heapster-5b9b6b6597-d5n67                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kube-system                kubernetes-dashboard-76479d66bb-j9bpg                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kube-system                metrics-server-79558444c6-9gx75                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ambassador-6f7cc986df-5qg87                                  200m (0%)     1 (3%)      100Mi (0%)       400Mi (0%)     2d
  kubeflow                   ambassador-6f7cc986df-lp6bq                                  200m (0%)     1 (3%)      100Mi (0%)       400Mi (0%)     2d
  kubeflow                   ambassador-6f7cc986df-m66w7                                  200m (0%)     1 (3%)      100Mi (0%)       400Mi (0%)     2d
  kubeflow                   argo-ui-db7cf456c-dlhlf                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   centraldashboard-79f6448bb7-x6q55                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   config-controller-6d84df4f66-6g85b                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   jupyter-0                                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   jupyter-web-app-78844bd57-lhsc2                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   katib-ui-bf44885cd-6bk4l                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   metacontroller-0                                             500m (1%)     4 (12%)     1Gi (1%)         4Gi (6%)       2d
  kubeflow                   minio-6d879f8d6c-k9sg9                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ml-pipeline-5dfc9cc665-4cvrz                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ml-pipeline-persistenceagent-5c5d669f5d-xg7lt                0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ml-pipeline-scheduledworkflow-84ddd9886d-hg7kr               0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ml-pipeline-ui-58c78c9ffb-dqlhd                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   ml-pipeline-viewer-controller-deployment-547bb45844-96xp9    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   mysql-58cfd7c97b-9vcjs                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   notebooks-controller-86c8944799-5x952                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   profiles-7896d9bd97-phd44                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   pytorch-operator-54484d9b6c-jpgp5                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   spartakus-volunteer-6798cc9878-kjpd8                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   studyjob-controller-58bccc4747-n4vcn                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   tf-job-dashboard-56564f6f99-9c6gb                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   tf-job-operator-6bfd5c7db8-qlctd                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-core-cfd9566b-mzsv8                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-core-rest-6c69cd9656-nkdwl                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-db-6885dbd6cb-twm5p                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-suggestion-bayesianoptimization-7ddbbd49b6-4tspd      0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-suggestion-grid-ccc744bfb-bwvhp                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-suggestion-hyperband-5bfbd98c78-snc7m                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   vizier-suggestion-random-f69bf84f4-4dx7p                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   workflow-controller-6866879d86-jhlsl                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d
  kubeflow                   wzy-0                                                        4 (12%)       0 (0%)      8Gi (12%)        0 (0%)         2d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                5550m (17%)   7 (21%)
  memory             9656Mi (15%)  5636Mi (8%)
  ephemeral-storage  0 (0%)        0 (0%)
Events:              <none>

[sashayakovtseva]

Can you please update sycri to the latest version?
Also node information says you are running docker. I think you pasted the wrong node info here. According to your pod yaml you schedule it to 10.18.127.1, but info is for 10.18.127.3.

[malixian]

sorry, it's my fault。and your suggestion is update sycri to 1.0.0-beta.6?current is 1.0.0-beta.5

Name:               10.18.127.1
Roles:              node
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=10.18.127.1
                    kubernetes.io/role=node
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 12 Jun 2019 16:20:03 +0800
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 26 Sep 2019 18:30:27 +0800   Fri, 21 Jun 2019 14:36:04 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 26 Sep 2019 18:30:27 +0800   Fri, 21 Jun 2019 14:36:04 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 26 Sep 2019 18:30:27 +0800   Fri, 21 Jun 2019 14:36:04 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 26 Sep 2019 18:30:27 +0800   Thu, 26 Sep 2019 16:29:11 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  10.18.127.1
  Hostname:    10.18.127.1
Capacity:
 cpu:                32
 ephemeral-storage:  511750Mi
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             65393136Ki
 pods:               110
Allocatable:
 cpu:                32
 ephemeral-storage:  482947890401
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             65290736Ki
 pods:               110
System Info:
 Machine ID:                 8b1ca9216f384e5c90f309b4af7066b1
 System UUID:                00000000-0000-0000-0000-AC1F6BAC1D10
 Boot ID:                    b3e7c019-5477-4a70-8e16-0b0ceac4da50
 Kernel Version:             3.10.0-957.21.2.el7.x86_64
 OS Image:                   CentOS Linux 7 (Core)
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  singularity://3.4.0-1
 Kubelet Version:            v1.13.5
 Kube-Proxy Version:         v1.13.5
PodCIDR:                     172.22.0.0/24
Non-terminated Pods:         (2 in total)
  Namespace                  Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                         ------------  ----------  ---------------  -------------  ---
  default                    image-service-deployment-655d89d94d-rfl5f    0 (0%)        0 (0%)      0 (0%)           0 (0%)         113m
  kube-system                calico-node-ps7bb                            250m (0%)     0 (0%)      0 (0%)           0 (0%)         106d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                250m (0%)  0 (0%)
  memory             0 (0%)     0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
Events:              <none>

[sashayakovtseva]

your suggestion is update sycri to 1.0.0-beta.6

Yes, there've been some fixes

[sashayakovtseva]

Also enabling debug logs will help us to understand what is wrong.
In sycri service add -v 6 option to sycri command and then restart services (assuming systemd is used):

$ sudo systemctl daemon-realod
$ sudo systemctl stop kubelet sycri 
$ sudo systemctl restart sycri kubelet

[malixian]

@sashayakovtseva unfortunately, it doesn't work.but i can see sycri detail execute information:

9月 27 11:16:46 comput1 sycri[189766]: DEBUG   [U=1000,P=1]       startup()                     oci runtime engine selected
9月 27 11:16:46 comput1 sycri[189766]: VERBOSE [U=1000,P=1]       startup()                     Execute stage 2
9月 27 11:16:46 comput1 sycri[189766]: DEBUG   [U=1000,P=1]       StageTwo()                    Entering stage 2
9月 27 11:16:46 comput1 sycri[189766]: I0927 11:16:46.874076  189766 sync.go:76] Received state 2 at /var/run/singularity/containers/2d3675bcb18c6c9aa0d34fe2018da97449838a8a7720a318f23f8da7de721b1a/sync.sock
9月 27 11:16:46 comput1 sycri[189766]: I0927 11:16:46.877823  189766 client_oci.go:125] Stream copying returned: context canceled
9月 27 11:16:46 comput1 sycri[189766]: I0927 11:16:46.989017  189766 container.go:288] Starting container 2d3675bcb18c6c9aa0d34fe2018da97449838a8a7720a318f23f8da7de721b1a
9月 27 11:16:46 comput1 sycri[189766]: I0927 11:16:46.989140  189766 client.go:87] Executing [singularity -d oci start 2d3675bcb18c6c9aa0d34fe2018da97449838a8a7720a318f23f8da7de721b1a]
9月 27 11:16:47 comput1 sycri[189766]: DEBUG   [U=0,P=175124]     createConfDir()               /root/.singularity already exists. Not creating.
9月 27 11:16:47 comput1 sycri[189766]: I0927 11:16:47.071929  189766 sync.go:76] Received state 4 at /var/run/singularity/containers/2d3675bcb18c6c9aa0d34fe2018da97449838a8a7720a318f23f8da7de721b1a/sync.sock
9月 27 11:16:47 comput1 sycri[189766]: E0927 11:16:47.079662  189766 main.go:276] /runtime.v1alpha2.RuntimeService/StartContainer
9月 27 11:16:47 comput1 sycri[189766]: Request: {"container_id":"2d3675bcb18c6c9aa0d34fe2018da97449838a8a7720a318f23f8da7de721b1a"}
9月 27 11:16:47 comput1 sycri[189766]: Response: null
9月 27 11:16:47 comput1 sycri[189766]: Error: rpc error: code = Internal desc = could not start container: unexpected container state: 4
9月 27 11:16:48 comput1 sycri[189766]: I0927 11:16:48.612509  189766 container_files.go:104] Removing bundle at /var/run/singularity/containers/1d632b4b2ce0d403c730b400469f5d663e8cc900992db8fd0ce0a1bf499e68a8/bundle
9月 27 11:16:48 comput1 sycri[189766]: I0927 11:16:48.657972  189766 container_files.go:118] Removing container base directory /var/run/singularity/containers/1d632b4b2ce0d403c730b400469f5d663e8cc900992db8fd0ce0a1bf499e68a8
9月 27 11:16:53 comput1 sycri[189766]: I0927 11:16:53.742925  189766 client_oci.go:166] Executing [singularity -d oci exec 620068bdf0dadcb5dc6409ab9ac27854a54745bebec5914d05f03debf2033c73 /bin/calico-node -bird-ready -felix-ready]
9月 27 11:17:03 comput1 sycri[189766]: I0927 11:17:03.742912  189766 client_oci.go:166] Executing [singularity -d oci exec 620068bdf0dadcb5dc6409ab9ac27854a54745bebec5914d05f03debf2033c73 /bin/calico-node -bird-ready -felix-ready]
9月 27 11:17:13 comput1 sycri[189766]: I0927 11:17:13.742929  189766 client_oci.go:166] Executing [singularity -d oci exec 620068bdf0dadcb5dc6409ab9ac27854a54745bebec5914d05f03debf2033c73 /bin/calico-node -bird-ready -felix-ready]
9月 27 11:17:23 comput1 sycri[189766]: I0927 11:17:23.742745  189766 client_oci.go:166] Executing [singularity -d oci exec 620068bdf0dadcb5dc6409ab9ac27854a54745bebec5914d05f03debf2033c73 /bin/calico-node -bird-ready -felix-ready]
9月 27 11:17:33 comput1 sycri[189766]: I0927 11:17:33.743061  189766 client_oci.go:166] Executing [singularity -d oci exec 620068bdf0dadcb5dc6409ab9ac27854a54745bebec5914d05f03debf2033c73 /bin/calico-node -bird-ready -felix-ready]

you can see , execution process doesn't have stage 3 and unexpected container state: 4,
can you explain what's wrong with it?

[sashayakovtseva]

Is that for sashayakovtseva/test/image-server? That image was built for amd64, so if you are scheduling it to arm it fails to start.