<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): GKE, GCE, volume timeout</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.4&quot;, GitCommit:&quot;3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-10-21T02:48:38Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.5&quot;, GitCommit:&quot;5a0a696437ad35c133c0c8493f7e9d22b0f9b81b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-10-29T01:32:42Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GKE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): GCI</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): whatever GCI uses</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>: Pods cannot start/restart because the volume will not mount. The volume cannot mount indefinitely (I have had pods in a ContainerCreating state for weeks in some cases).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="21s       2m        2         camlistored-2824072653-3p1t7   Pod                                 Warning   FailedMount               {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   Unable to mount volumes for pod &quot;camlistored-2824072653-3p1t7_prod(bd1cbbe3-b380-11e6-9d1b-42010af000ba)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;camlistored-2824072653-3p1t7&quot;/&quot;prod&quot;. list of unattached/unmounted volumes=[camlistore-blobs]
21s       2m        2         camlistored-2824072653-3p1t7   Pod                                 Warning   FailedSync                {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;camlistored-2824072653-3p1t7&quot;/&quot;prod&quot;. list of unattached/unmounted volumes=[camlistore-blobs]"><pre class="notranslate"><code class="notranslate">21s       2m        2         camlistored-2824072653-3p1t7   Pod                                 Warning   FailedMount               {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   Unable to mount volumes for pod "camlistored-2824072653-3p1t7_prod(bd1cbbe3-b380-11e6-9d1b-42010af000ba)": timeout expired waiting for volumes to attach/mount for pod "camlistored-2824072653-3p1t7"/"prod". list of unattached/unmounted volumes=[camlistore-blobs]
21s       2m        2         camlistored-2824072653-3p1t7   Pod                                 Warning   FailedSync                {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "camlistored-2824072653-3p1t7"/"prod". list of unattached/unmounted volumes=[camlistore-blobs]
</code></pre></div>
<p dir="auto">I sometimes notice other errors when errors first start occurring. Here is one example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="15s       4m        11        camlistored-2824072653-3p1t7   Pod                 Warning   FailedMount   {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   MountVolume.SetUp failed for volume &quot;kubernetes.io/gce-pd/camlistore-blobs-prod&quot; (spec.Name: &quot;camlistore-blobs-prod&quot;) pod &quot;bd1cbbe3-b380-11e6-9d1b-42010af000ba&quot; (UID: &quot;bd1cbbe3-b380-11e6-9d1b-42010af000ba&quot;) with: mount failed: exit status 32
Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod /var/lib/kubelet/pods/bd1cbbe3-b380-11e6-9d1b-42010af000ba/volumes/kubernetes.io~gce-pd/camlistore-blobs-prod  [remount]
Output: mount: cannot remount /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod read-write, is write-protected"><pre class="notranslate"><code class="notranslate">15s       4m        11        camlistored-2824072653-3p1t7   Pod                 Warning   FailedMount   {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   MountVolume.SetUp failed for volume "kubernetes.io/gce-pd/camlistore-blobs-prod" (spec.Name: "camlistore-blobs-prod") pod "bd1cbbe3-b380-11e6-9d1b-42010af000ba" (UID: "bd1cbbe3-b380-11e6-9d1b-42010af000ba") with: mount failed: exit status 32
Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod /var/lib/kubelet/pods/bd1cbbe3-b380-11e6-9d1b-42010af000ba/volumes/kubernetes.io~gce-pd/camlistore-blobs-prod  [remount]
Output: mount: cannot remount /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod read-write, is write-protected
</code></pre></div>
<p dir="auto">Deleting and recreating a deployment sometimes works to allow me to remount the volume but it's rare.</p>
<p dir="auto"><strong>What you expected to happen</strong>: I expect that volumes will be able to be re-mounted to pods within a reasonable time period. e.g. a few minutes.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">I don't know the specific circumstances that this happens but I notice that this happens often when a container with the volume attached crashes. The container will crash and cannot be restarted because the volume will not mount.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">This happens <em>very frequently</em> for certain types of pods that use the disk often like camlistore or MySQL when the pod is stopped and restarted, via deleting the pod, cluster upgrades etc. It's extremely frustrating.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>