Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod. using sample-prometheus-instance.yaml

[bamb00]

Hi,

I'm getting the error Unable to mount volumes for pod error from the sample-prometheus-instance.yaml. No change was made in the yaml file.

Was wondering anyone encounters the same error?

https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/sample-prometheus-instance.yaml

   Unable to mount volumes for pod "prometheus-sample-metrics-prom-0_default(469b4880-0a9b-11e8-a29b-0800272637bb)": timeout expired waiting for volumes to attach/mount for pod "default"/"prometheus-sample-metrics-prom-0". list of unattached/unmounted volumes=[prometheus-sample-metrics-prom-db]

journalctl:

   Feb 05 20:05:53 minikube kubelet[3704]: E0205 20:05:53.570497    3704 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/rbd/[10.97.152.94:6790 10.104.200.62:6790 10.111.171.163:6790]:k8s-dynamic-pvc-ea65c0b0-0a99-11e8-a29b-0800272637bb-ea742ba1-0a99-11e8-9a4d-0242ac110004\"" failed. No retries permitted until 2018-02-05 20:07:55.570463699 +0000 UTC m=+40710.307964873 (durationBeforeRetry 2m2s). Error: "MountVolume.WaitForAttach failed for volume \"pvc-ea65c0b0-0a99-11e8-a29b-0800272637bb\" (UniqueName: \"kubernetes.io/rbd/[10.97.152.94:6790 10.104.200.62:6790 10.111.171.163:6790]:k8s-dynamic-pvc-ea65c0b0-0a99-11e8-a29b-0800272637bb-ea742ba1-0a99-11e8-9a4d-0242ac110004\") pod \"prometheus-sample-metrics-prom-0\" (UID: \"38ed9d87-0aa9-11e8-a29b-0800272637bb\") : error: executable file not found in $PATH, rbd output: "
   Feb 05 20:05:53 minikube kubelet[3704]: I0205 20:05:53.368544    3704 operation_generator.go:446] MountVolume.WaitForAttach entering for volume "pvc-ea65c0b0-0a99-11e8-a29b-0800272637bb" (UniqueName: "kubernetes.io/rbd/[10.97.152.94:6790 10.104.200.62:6790 10.111.171.163:6790]:k8s-dynamic-pvc-ea65c0b0-0a99-11e8-a29b-0800272637bb-ea742ba1-0a99-11e8-9a4d-0242ac110004") pod "prometheus-sample-metrics-prom-0" (UID: "38ed9d87-0aa9-11e8-a29b-0800272637bb") DevicePath ""
   Feb 05 20:05:02 minikube kubelet[3704]: E0205 20:05:02.532512    3704 pod_workers.go:186] Error syncing pod 38ed9d87-0aa9-11e8-a29b-0800272637bb ("prometheus-sample-metrics-prom-0_default(38ed9d87-0aa9-11e8-a29b-0800272637bb)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"prometheus-sample-metrics-prom-0". list of unattached/unmounted volumes=[prometheus-sample-metrics-prom-db]

Env Info:
Minikube: v1.9.1
OS: Linux minikube 4.9.13 #1 SMP Thu Oct 19 17:14:00 UTC 2017 x86_64 GNU/Linux

Thanks in Advance.

[NoahLu1105]

do you solve it ? i meat same problem in my k8s cluster with ceph rbd