<p dir="auto">I'm running k8s on AWS.  <code class="notranslate">kube-up.sh</code> works, and I can run a simple Rails app (controller and service).  Everything is great, but when I go back the next day to access the app or load a new version the nodes have degraded and are all <code class="notranslate">NotReady</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get nodes
NAME                                         LABELS                                                              STATUS
ip-172-20-0-196.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-196.us-west-2.compute.internal   NotReady
ip-172-20-0-197.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-197.us-west-2.compute.internal   NotReady
ip-172-20-0-198.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-198.us-west-2.compute.internal   NotReady
ip-172-20-0-199.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-199.us-west-2.compute.internal   NotReady"><pre class="notranslate"><code class="notranslate">$ kubectl get nodes
NAME                                         LABELS                                                              STATUS
ip-172-20-0-196.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-196.us-west-2.compute.internal   NotReady
ip-172-20-0-197.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-197.us-west-2.compute.internal   NotReady
ip-172-20-0-198.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-198.us-west-2.compute.internal   NotReady
ip-172-20-0-199.us-west-2.compute.internal   kubernetes.io/hostname=ip-172-20-0-199.us-west-2.compute.internal   NotReady
</code></pre></div>
<p dir="auto">The pods are <code class="notranslate">Pending</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
chronic-controller-16uic       0/1       Pending   0          6h
chronic-controller-6x025       0/1       Pending   0          1h
chronic-controller-x7owf       0/1       Pending   0          6h
private-registry-pbvdi         0/1       Pending   0          6h
tripplanner-controller-3q6km   0/1       Pending   0          6h
tripplanner-controller-bhwwy   0/1       Pending   0          1h
tripplanner-controller-fdep8   0/1       Pending   0          6h"><pre class="notranslate"><code class="notranslate">$ kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
chronic-controller-16uic       0/1       Pending   0          6h
chronic-controller-6x025       0/1       Pending   0          1h
chronic-controller-x7owf       0/1       Pending   0          6h
private-registry-pbvdi         0/1       Pending   0          6h
tripplanner-controller-3q6km   0/1       Pending   0          6h
tripplanner-controller-bhwwy   0/1       Pending   0          1h
tripplanner-controller-fdep8   0/1       Pending   0          6h
</code></pre></div>
<p dir="auto">Here are the recent events:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get events
FIRSTSEEN                         LASTSEEN                          COUNT     NAME                                         KIND      SUBOBJECT   REASON             SOURCE                                                 MESSAGE
Sat, 12 Sep 2015 22:44:10 -0500   Tue, 15 Sep 2015 15:04:08 -0500   44        ip-172-20-0-199.us-west-2.compute.internal   Node                  NodeNotReady       {kubelet ip-172-20-0-199.us-west-2.compute.internal}   Node ip-172-20-0-199.us-west-2.compute.internal status is now: NodeNotReady
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:41 -0500   7531      chronic-controller-ljzn6                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:41 -0500   7757      private-registry-ysjfj                       Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7284      tripplanner-controller-7b0yr                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7413      tripplanner-controller-s86wz                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7374      chronic-controller-f2nrx                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7447      chronic-controller-j4538                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 01:42:38 -0500   Tue, 15 Sep 2015 15:22:42 -0500   9303      tripplanner-controller-04ztj                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods"><pre class="notranslate"><code class="notranslate">$ kubectl get events
FIRSTSEEN                         LASTSEEN                          COUNT     NAME                                         KIND      SUBOBJECT   REASON             SOURCE                                                 MESSAGE
Sat, 12 Sep 2015 22:44:10 -0500   Tue, 15 Sep 2015 15:04:08 -0500   44        ip-172-20-0-199.us-west-2.compute.internal   Node                  NodeNotReady       {kubelet ip-172-20-0-199.us-west-2.compute.internal}   Node ip-172-20-0-199.us-west-2.compute.internal status is now: NodeNotReady
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:41 -0500   7531      chronic-controller-ljzn6                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:41 -0500   7757      private-registry-ysjfj                       Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7284      tripplanner-controller-7b0yr                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7413      tripplanner-controller-s86wz                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7374      chronic-controller-f2nrx                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 09:24:35 -0500   Tue, 15 Sep 2015 15:22:42 -0500   7447      chronic-controller-j4538                     Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
Mon, 14 Sep 2015 01:42:38 -0500   Tue, 15 Sep 2015 15:22:42 -0500   9303      tripplanner-controller-04ztj                 Pod                   failedScheduling   {scheduler }                                           no nodes available to schedule pods
</code></pre></div>
<p dir="auto">I logged into the master and one of the minions to look at <code class="notranslate">/var/log/kubelet.log</code> but that file doesn't exist on those hosts.  Trying to ssh into one of the AWS instances is also spotty for me, but the EC2 instances all claim to be up and healthy.</p>
<p dir="auto">I have killed everything with <code class="notranslate">kube-down.sh</code> and built it all again.  Everything worked for the afternoon and then was down again the next morning.  I don't know what else to look at or do.  I'm happy to look at things.  I'm <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/barrettclark/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/barrettclark">@barrettclark</a> in the slack channel.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.6</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Google, Container-Optimized OS, beta 55-8872.47.0, Kernel: ChromiumOS-4.4.4 Kubernetes: 1.4.5 Docker: 1.11.2</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-warpdrive-default-pool-eae1f3b0-spgr 4.4.21+ <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Thu Nov 10 21:43:53 PST 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
A a persistent disk was created via:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gcloud compute disks create warpdrive-jenkins-home --type pd-standard"><pre class="notranslate"><code class="notranslate">gcloud compute disks create warpdrive-jenkins-home --type pd-standard
</code></pre></div>
<p dir="auto">This persistent disk is referenced in a K8s Deployment (yaml).  The Pod failed on startup, due to FailedMount followed by FailedSync.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe pods --namespace=jenkins | tail 
Events:
  FirstSeen	LastSeen	Count	From	SubobjectPath	Type	Reason	Message
  ---------	--------	-----	----	-------------	--------	------	-------
  5m	5m	1	{default-scheduler }	Normal	Scheduled	Successfully assigned jenkins-master-3888267560-x2rpc to gke-warpdrive-default-pool-af4bf63e-qjrx
  3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedMount	Unable to mount volumes for pod &quot;jenkins-master-3888267560-x2rpc_jenkins(5f752595-b01a-11e6-972b-42010a8e0029)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;jenkins-master-3888267560-x2rpc&quot;/&quot;jenkins&quot;. list of unattached/unmounted volumes=[jenkins-home]
  3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;jenkins-master-3888267560-x2rpc&quot;/&quot;jenkins&quot;. list of unattached/unmounted volumes=[jenkins-home]"><pre class="notranslate"><code class="notranslate">$ kubectl describe pods --namespace=jenkins | tail 
Events:
  FirstSeen	LastSeen	Count	From	SubobjectPath	Type	Reason	Message
  ---------	--------	-----	----	-------------	--------	------	-------
  5m	5m	1	{default-scheduler }	Normal	Scheduled	Successfully assigned jenkins-master-3888267560-x2rpc to gke-warpdrive-default-pool-af4bf63e-qjrx
  3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedMount	Unable to mount volumes for pod "jenkins-master-3888267560-x2rpc_jenkins(5f752595-b01a-11e6-972b-42010a8e0029)": timeout expired waiting for volumes to attach/mount for pod "jenkins-master-3888267560-x2rpc"/"jenkins". list of unattached/unmounted volumes=[jenkins-home]
  3m	1m	2	{kubelet gke-warpdrive-default-pool-af4bf63e-qjrx}	Warning	FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "jenkins-master-3888267560-x2rpc"/"jenkins". list of unattached/unmounted volumes=[jenkins-home]
</code></pre></div>
<p dir="auto">Interestingly, the persistent disk was actually attached:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ gcloud compute disks describe warpdrive-jenkins-home | tail 
lastAttachTimestamp: '2016-11-21T10:43:21.433-08:00'
lastDetachTimestamp: '2016-11-21T10:37:20.688-08:00'
name: warpdrive-jenkins-home
selfLink: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/disks/warpdrive-jenkins-home
sizeGb: '1000'
status: READY
type: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/diskTypes/pd-standard
users:
- https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/instances/gke-warpdrive-default-pool-af4bf63e-qjrx
zone: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b"><pre class="notranslate"><code class="notranslate">$ gcloud compute disks describe warpdrive-jenkins-home | tail 
lastAttachTimestamp: '2016-11-21T10:43:21.433-08:00'
lastDetachTimestamp: '2016-11-21T10:37:20.688-08:00'
name: warpdrive-jenkins-home
selfLink: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/disks/warpdrive-jenkins-home
sizeGb: '1000'
status: READY
type: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/diskTypes/pd-standard
users:
- https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b/instances/gke-warpdrive-default-pool-af4bf63e-qjrx
zone: https://www.googleapis.com/compute/v1/projects/symphony-gce-dev/zones/us-east1-b
</code></pre></div>
<p dir="auto">I attempted to recreate , but the issue persisted across all attempts.</p>
<p dir="auto">Worth noting is that the issue is avoided if the persistent disk is defined by:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v2.tar.gz
gcloud compute disks create jenkins-home --image jenkins-home-image"><pre class="notranslate"><code class="notranslate">gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v2.tar.gz
gcloud compute disks create jenkins-home --image jenkins-home-image
</code></pre></div>
<p dir="auto">Also worth noting is that I was able to ssh into a node and manually mount, write some files, unmount, then have K8s auto detach/attach via deletion/creation of Deployments, and then manually mount and verify that written files were there.</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
I expected the persistent disk to attach and mount with no issue.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ul dir="auto">
<li>gcloud container clusters create warpdrive --machine-type n1-standard-4 --num-nodes 4 --subnetwork sym-infra-dev-guse1 --enable-autoscaling --min-nodes 4 --max-nodes 20 --scopes "<a href="https://www.googleapis.com/auth/projecthosting,storage-rw" rel="nofollow">https://www.googleapis.com/auth/projecthosting,storage-rw</a>"</li>
<li>gcloud compute disks create warpdrive-jenkins-home --type pd-standard</li>
<li>kubectl create ns jenkins</li>
<li>create Deployment with:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
      volumes:
      - name: jenkins-home
        gcePersistentDisk:
          pdName: warpdrive-jenkins-home
          fsType: ext4
          partition: 1"><pre class="notranslate"><code class="notranslate">    volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
      volumes:
      - name: jenkins-home
        gcePersistentDisk:
          pdName: warpdrive-jenkins-home
          fsType: ext4
          partition: 1
</code></pre></div>
<ul dir="auto">
<li>watch in horror as the pod fails</li>
</ul>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
We're desperate</p>