<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pods
(...)
fluentd-elasticsearch-kubernetes-minion-7ewp                                                                                             kubernetes-minion-7ewp/                 &lt;none&gt;                                                                        Pending   Less than a second   
                                                            fluentd-elasticsearch   gcr.io/google_containers/fluentd-elasticsearch:1.5                                                                                                                                   
fluentd-elasticsearch-kubernetes-minion-aizq                                                                                             kubernetes-minion-aizq/                 &lt;none&gt;                                                                        Pending   1 seconds   
                                                            fluentd-elasticsearch   gcr.io/google_containers/fluentd-elasticsearch:1.5                                                                                                                                 
(...)  "><pre class="notranslate"><code class="notranslate">$ kubectl get pods
(...)
fluentd-elasticsearch-kubernetes-minion-7ewp                                                                                             kubernetes-minion-7ewp/                 &lt;none&gt;                                                                        Pending   Less than a second   
                                                            fluentd-elasticsearch   gcr.io/google_containers/fluentd-elasticsearch:1.5                                                                                                                                   
fluentd-elasticsearch-kubernetes-minion-aizq                                                                                             kubernetes-minion-aizq/                 &lt;none&gt;                                                                        Pending   1 seconds   
                                                            fluentd-elasticsearch   gcr.io/google_containers/fluentd-elasticsearch:1.5                                                                                                                                 
(...)  
</code></pre></div>
<p dir="auto">The status is always pending with the created time equal 1s or less than 1s.</p>
<p dir="auto">The logs from kubelet show these messages at about a second interval:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E0522 01:10:23.012618    2893 kubelet.go:1074] Deleting mirror pod &quot;fluentd-elasticsearch-kubernetes-minion-aizq_default&quot; because it is outdated
W0522 01:10:24.217263    2893 status_manager.go:60] Failed to updated pod status: error updating status for pod &quot;fluentd-elasticsearch-kubernetes-minion-aizq&quot;: pods &quot;fluentd-elasticsearch-kubernetes-minion-aizq&quot; not found"><pre class="notranslate"><code class="notranslate">E0522 01:10:23.012618    2893 kubelet.go:1074] Deleting mirror pod "fluentd-elasticsearch-kubernetes-minion-aizq_default" because it is outdated
W0522 01:10:24.217263    2893 status_manager.go:60] Failed to updated pod status: error updating status for pod "fluentd-elasticsearch-kubernetes-minion-aizq": pods "fluentd-elasticsearch-kubernetes-minion-aizq" not found
</code></pre></div>
<p dir="auto">Possibly related to/duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="167374488" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29548" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/29548/hovercard" href="https://github.com/kubernetes/kubernetes/issues/29548">#29548</a></p>
<p dir="auto">I'm running Kubernetes 1.3.0 and having trouble mounting EBS volumes to pods. The volumes seem to be attached to the minion, but not mounted.</p>
<p dir="auto"><code class="notranslate">xvdba</code> device is the EBS volume I'm trying to mount:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ip-10-72-4-178 ~ # lsblk
NAME    MAJ:MIN   RM   SIZE RO TYPE MOUNTPOINT
xvda    202:0      0   200G  0 disk
|-xvda1 202:1      0   128M  0 part /boot
|-xvda2 202:2      0     2M  0 part
|-xvda3 202:3      0     1G  0 part /usr
|-xvda4 202:4      0     1G  0 part
|-xvda6 202:6      0   128M  0 part /usr/share/oem
|-xvda7 202:7      0    64M  0 part
`-xvda9 202:9      0 197.7G  0 part /
xvdba   202:13312  0     5G  0 disk"><pre class="notranslate"><code class="notranslate">ip-10-72-4-178 ~ # lsblk
NAME    MAJ:MIN   RM   SIZE RO TYPE MOUNTPOINT
xvda    202:0      0   200G  0 disk
|-xvda1 202:1      0   128M  0 part /boot
|-xvda2 202:2      0     2M  0 part
|-xvda3 202:3      0     1G  0 part /usr
|-xvda4 202:4      0     1G  0 part
|-xvda6 202:6      0   128M  0 part /usr/share/oem
|-xvda7 202:7      0    64M  0 part
`-xvda9 202:9      0 197.7G  0 part /
xvdba   202:13312  0     5G  0 disk
</code></pre></div>
<p dir="auto">The RC definition:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="volumes:
  - awsElasticBlockStore:
      fsType: ext4
      volumeID: aws://us-west-2a/vol-cee37b47
    name: jenkins-data"><pre class="notranslate"><code class="notranslate">volumes:
  - awsElasticBlockStore:
      fsType: ext4
      volumeID: aws://us-west-2a/vol-cee37b47
    name: jenkins-data
</code></pre></div>
<p dir="auto">From <code class="notranslate">kube-controller-manager</code> log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Jul 26 15:15:06 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:06.743994    2158 reconciler.go:114] Started DetachVolume for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; from node &quot;ip-10-72-4-178.us-west-2.compute.internal&quot;
Jul 26 15:15:06 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:06.747351    2158 operation_executor.go:546] Verified volume is safe to detach for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; (spec.Name: &quot;jenkins-data&quot;) from node &quot;ip-10-72-4-178.us-west-2.compute.internal&quot;.
Jul 26 15:15:14 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: E0726 15:15:14.047011    2158 aws.go:1136] endAttaching on non-allocated device
Jul 26 15:15:14 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:14.047042    2158 operation_executor.go:565] DetachVolume.Detach succeeded for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; (spec.Name: &quot;jenkins-data&quot;) from node &quot;ip-10-72-4-178.us-west-2.compute.internal&quot;.
Jul 26 15:15:21 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:21.176639    2158 reconciler.go:169] Started AttachVolume for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; to node &quot;ip-10-72-4-178.us-west-2.compute.internal&quot;
Jul 26 15:15:25 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:25.022432    2158 operation_executor.go:454] AttachVolume.Attach succeeded for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; (spec.Name: &quot;jenkins-data&quot;) from node &quot;ip-10-72-4-178.us-west-2.compute.internal&quot;."><pre class="notranslate"><code class="notranslate">Jul 26 15:15:06 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:06.743994    2158 reconciler.go:114] Started DetachVolume for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" from node "ip-10-72-4-178.us-west-2.compute.internal"
Jul 26 15:15:06 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:06.747351    2158 operation_executor.go:546] Verified volume is safe to detach for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" (spec.Name: "jenkins-data") from node "ip-10-72-4-178.us-west-2.compute.internal".
Jul 26 15:15:14 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: E0726 15:15:14.047011    2158 aws.go:1136] endAttaching on non-allocated device
Jul 26 15:15:14 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:14.047042    2158 operation_executor.go:565] DetachVolume.Detach succeeded for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" (spec.Name: "jenkins-data") from node "ip-10-72-4-178.us-west-2.compute.internal".
Jul 26 15:15:21 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:21.176639    2158 reconciler.go:169] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" to node "ip-10-72-4-178.us-west-2.compute.internal"
Jul 26 15:15:25 ip-10-72-1-244.us-west-2.compute.internal kube-controller-manager[2158]: I0726 15:15:25.022432    2158 operation_executor.go:454] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" (spec.Name: "jenkins-data") from node "ip-10-72-4-178.us-west-2.compute.internal".
</code></pre></div>
<p dir="auto">From <code class="notranslate">kube-kubelet</code> log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Jul 26 15:14:45 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: I0726 15:14:45.075708    1309 reconciler.go:179] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; (spec.Name: &quot;jenkins-data&quot;) pod &quot;2a44c2b7-5343-11e6-ae02-06e2e967e82b&quot; (UID: &quot;2a44c2b7-5343-11e6-ae02-06e2e967e82b&quot;)
Jul 26 15:14:45 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:14:45.080193    1309 goroutinemap.go:155] Operation for &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; failed. No retries permitted until 2016-07-26 15:16:45.080187905 +0000 UTC (durationBeforeRetry 2m0s). error: VerifyControllerAttachedVolume failed fetching node from API server. Volume &quot;kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47&quot; (spec.Name: &quot;jenkins-data&quot;) pod &quot;2a44c2b7-5343-11e6-ae02-06e2e967e82b&quot; (UID: &quot;2a44c2b7-5343-11e6-ae02-06e2e967e82b&quot;). Error: nodes &quot;10.72.4.178&quot; not found.
Jul 26 15:15:01 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:01.445341    1309 file.go:53] Unable to read config path &quot;/etc/kubernetes/manifests&quot;: path does not exist, ignoring
Jul 26 15:15:04 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:04.906056    1309 kubelet.go:1930] Unable to mount volumes for pod &quot;jenkins-6xpbb_default(2a44c2b7-5343-11e6-ae02-06e2e967e82b)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;jenkins-6xpbb&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[jenkins-data]; skipping pod
Jul 26 15:15:04 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:04.906089    1309 pod_workers.go:183] Error syncing pod 2a44c2b7-5343-11e6-ae02-06e2e967e82b, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;jenkins-6xpbb&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[jenkins-data]"><pre class="notranslate"><code class="notranslate">Jul 26 15:14:45 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: I0726 15:14:45.075708    1309 reconciler.go:179] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" (spec.Name: "jenkins-data") pod "2a44c2b7-5343-11e6-ae02-06e2e967e82b" (UID: "2a44c2b7-5343-11e6-ae02-06e2e967e82b")
Jul 26 15:14:45 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:14:45.080193    1309 goroutinemap.go:155] Operation for "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" failed. No retries permitted until 2016-07-26 15:16:45.080187905 +0000 UTC (durationBeforeRetry 2m0s). error: VerifyControllerAttachedVolume failed fetching node from API server. Volume "kubernetes.io/aws-ebs/aws://us-west-2a/vol-cee37b47" (spec.Name: "jenkins-data") pod "2a44c2b7-5343-11e6-ae02-06e2e967e82b" (UID: "2a44c2b7-5343-11e6-ae02-06e2e967e82b"). Error: nodes "10.72.4.178" not found.
Jul 26 15:15:01 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:01.445341    1309 file.go:53] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jul 26 15:15:04 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:04.906056    1309 kubelet.go:1930] Unable to mount volumes for pod "jenkins-6xpbb_default(2a44c2b7-5343-11e6-ae02-06e2e967e82b)": timeout expired waiting for volumes to attach/mount for pod "jenkins-6xpbb"/"default". list of unattached/unmounted volumes=[jenkins-data]; skipping pod
Jul 26 15:15:04 ip-10-72-4-178.us-west-2.compute.internal kubelet[1309]: E0726 15:15:04.906089    1309 pod_workers.go:183] Error syncing pod 2a44c2b7-5343-11e6-ae02-06e2e967e82b, skipping: timeout expired waiting for volumes to attach/mount for pod "jenkins-6xpbb"/"default". list of unattached/unmounted volumes=[jenkins-data]
</code></pre></div>