<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">ebs aws kubelet</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">1.4.5</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Ubuntu-16.04</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.4.0-43-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35518434" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/63" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/63/hovercard" href="https://github.com/kubernetes/kubernetes/pull/63">#63</a>-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: custom</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ol dir="auto">
<li>Node has a pod with AWS EBS attached</li>
<li>Docker restarts, kubelet restarts</li>
<li>Pod goes to <code class="notranslate">Error</code> status, with messages</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" 1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   Warning         FailedMount     Unable to moun
t volumes for pod &quot;registry-341515849-b1f4k_default(123242a0-a75b-11e6-a3fb-0a8cd5340591)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;regis
try-341515849-b1f4k&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[storage]
  1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   Warning         FailedSync      Error syncing
pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;registry-341515849-b1f4k&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[storage
]"><pre class="notranslate"><code class="notranslate"> 1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   Warning         FailedMount     Unable to moun
t volumes for pod "registry-341515849-b1f4k_default(123242a0-a75b-11e6-a3fb-0a8cd5340591)": timeout expired waiting for volumes to attach/mount for pod "regis
try-341515849-b1f4k"/"default". list of unattached/unmounted volumes=[storage]
  1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   Warning         FailedSync      Error syncing
pod, skipping: timeout expired waiting for volumes to attach/mount for pod "registry-341515849-b1f4k"/"default". list of unattached/unmounted volumes=[storage
]
</code></pre></div>
<ol start="4" dir="auto">
<li>EBS volume is correctly attached and mounted to the node, <code class="notranslate">volumesAttached</code> and <code class="notranslate">volumesInUse</code> is correct on the node. Controller manager detects no problems, and doesn't do anything.</li>
<li>Pod never recovers. Manual unmount/detach/pod delete is required to recover.</li>
</ol>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
After a kubelet restart, pod should be healthy and pod sync succeeds.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Create a pod with attached EBS</li>
<li>Restart kubelet</li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">I think this is a regression from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="179642080" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33616" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/33616/hovercard" href="https://github.com/kubernetes/kubernetes/pull/33616">#33616</a>, since it worked for me in 1.3. But not sure, need to dig further.</p>
<p dir="auto">Need to add reconciler logic to the service controller sync loop to allow cloud provider packages to reconcile healthcheck and target pools created/needed by the service loadbalancer feature.</p>