<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">timeout expired waiting for volumes to attach/mount gcePersistentDisk readOnly</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">Bug report</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"windows/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GKE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="PRETTY_NAME=&quot;Debian GNU/Linux 7 (wheezy)&quot;
NAME=&quot;Debian GNU/Linux&quot;
VERSION_ID=&quot;7&quot;
VERSION=&quot;7 (wheezy)&quot;
ID=debian
ANSI_COLOR=&quot;1;31&quot;
HOME_URL=&quot;http://www.debian.org/&quot;
SUPPORT_URL=&quot;http://www.debian.org/support/&quot;
BUG_REPORT_URL=&quot;http://bugs.debian.org/&quot;"><pre class="notranslate">PRETTY_NAME=<span class="pl-s"><span class="pl-pds">"</span>Debian GNU/Linux 7 (wheezy)<span class="pl-pds">"</span></span>
NAME=<span class="pl-s"><span class="pl-pds">"</span>Debian GNU/Linux<span class="pl-pds">"</span></span>
VERSION_ID=<span class="pl-s"><span class="pl-pds">"</span>7<span class="pl-pds">"</span></span>
VERSION=<span class="pl-s"><span class="pl-pds">"</span>7 (wheezy)<span class="pl-pds">"</span></span>
ID=debian
ANSI_COLOR=<span class="pl-s"><span class="pl-pds">"</span>1;31<span class="pl-pds">"</span></span>
HOME_URL=<span class="pl-s"><span class="pl-pds">"</span>http://www.debian.org/<span class="pl-pds">"</span></span>
SUPPORT_URL=<span class="pl-s"><span class="pl-pds">"</span>http://www.debian.org/support/<span class="pl-pds">"</span></span>
BUG_REPORT_URL=<span class="pl-s"><span class="pl-pds">"</span>http://bugs.debian.org/<span class="pl-pds">"</span></span></pre></div>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-fatih-small-pool-59881027-k909 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">readOnly persistent disk won't mount multiple times</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Can be mounted multiple times</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Create 3 RCs (with 1 replica each) that mount the same persistent disk as read only.</p>
<p dir="auto">The first pod always succeeds mounting.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get po
NAME              READY     STATUS              RESTARTS   AGE
mongo-express     1/1       Running             0          2h
mongo-rc0-rpah5   1/1       Running             0          30m
mongo-rc1-xkcyy   1/1       Running             0          17m
mongo-rc2-aktox   0/1       ContainerCreating   0          30m"><pre class="notranslate"><code class="notranslate">kubectl get po
NAME              READY     STATUS              RESTARTS   AGE
mongo-express     1/1       Running             0          2h
mongo-rc0-rpah5   1/1       Running             0          30m
mongo-rc1-xkcyy   1/1       Running             0          17m
mongo-rc2-aktox   0/1       ContainerCreating   0          30m
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl describe po mongo-rc0-rpah5
Name:           mongo-rc0-rpah5
Namespace:      default
Node:           gke-fatih-small-pool-59881027-k909/10.142.0.5
Start Time:     Tue, 23 Aug 2016 05:01:48 +0700
Labels:         instance=fatih0
                name=mongo-node0
Status:         Running
IP:             10.60.2.5
Controllers:    ReplicationController/mongo-rc0
Containers:
  mongo-node0:
    Container ID:       docker://2845af5d815ff62505284ee8ef22bb5be3fa7f276e00d9347c9ce0e726e45d2f
    Image:              mongo
    Image ID:           docker://sha256:af52553e1c34b3ec48a2e50cf73a1eed1fc6d2fd2b0d3d73d7397c8d6341551f
    Port:               27017/TCP
    Command:
      mongod
      --replSet
      bippo
      --storageEngine
      wiredTiger
      --keyFile
      /etc/mongo-conf/mongo.keyfile
    Requests:
      cpu:                      100m
    State:                      Running
      Started:                  Tue, 23 Aug 2016 05:02:06 +0700
    Ready:                      True
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
Volumes:
  mongo-conf:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongo-conf
    FSType:     ext4
    Partition:  0
    ReadOnly:   true
  mongo-persistent-storage0:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongodb-disk0
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-i0lox:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-i0lox
QoS Tier:       Burstable
Events:
  FirstSeen     LastSeen        Count   From                                            SubobjectPath
Type            Reason          Message
  ---------     --------        -----   ----                                            -------------
--------        ------          -------
  33m           33m             1       {default-scheduler }                                                    Normal
        Scheduled       Successfully assigned mongo-rc0-rpah5 to gke-fatih-small-pool-59881027-k909
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Pulling         pulling image &quot;mongo&quot;
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Pulled          Successfully pulled image &quot;mongo&quot;
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Created         Created container with docker id 2845af5d815f
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Started         Started container with docker id 2845af5d815f"><pre class="notranslate"><code class="notranslate">kubectl describe po mongo-rc0-rpah5
Name:           mongo-rc0-rpah5
Namespace:      default
Node:           gke-fatih-small-pool-59881027-k909/10.142.0.5
Start Time:     Tue, 23 Aug 2016 05:01:48 +0700
Labels:         instance=fatih0
                name=mongo-node0
Status:         Running
IP:             10.60.2.5
Controllers:    ReplicationController/mongo-rc0
Containers:
  mongo-node0:
    Container ID:       docker://2845af5d815ff62505284ee8ef22bb5be3fa7f276e00d9347c9ce0e726e45d2f
    Image:              mongo
    Image ID:           docker://sha256:af52553e1c34b3ec48a2e50cf73a1eed1fc6d2fd2b0d3d73d7397c8d6341551f
    Port:               27017/TCP
    Command:
      mongod
      --replSet
      bippo
      --storageEngine
      wiredTiger
      --keyFile
      /etc/mongo-conf/mongo.keyfile
    Requests:
      cpu:                      100m
    State:                      Running
      Started:                  Tue, 23 Aug 2016 05:02:06 +0700
    Ready:                      True
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
Volumes:
  mongo-conf:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongo-conf
    FSType:     ext4
    Partition:  0
    ReadOnly:   true
  mongo-persistent-storage0:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongodb-disk0
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-i0lox:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-i0lox
QoS Tier:       Burstable
Events:
  FirstSeen     LastSeen        Count   From                                            SubobjectPath
Type            Reason          Message
  ---------     --------        -----   ----                                            -------------
--------        ------          -------
  33m           33m             1       {default-scheduler }                                                    Normal
        Scheduled       Successfully assigned mongo-rc0-rpah5 to gke-fatih-small-pool-59881027-k909
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Pulling         pulling image "mongo"
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Pulled          Successfully pulled image "mongo"
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Created         Created container with docker id 2845af5d815f
  33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0}
Normal          Started         Started container with docker id 2845af5d815f
</code></pre></div>
<p dir="auto">The second pod is intermittent. The third pod (below) always fails.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl describe po mongo-rc2-aktox
Name:           mongo-rc2-aktox
Namespace:      default
Node:           gke-fatih-small-pool-59881027-h46g/10.142.0.6
Start Time:     Tue, 23 Aug 2016 05:01:49 +0700
Labels:         instance=fatih2
                name=mongo-node2
Status:         Pending
IP:
Controllers:    ReplicationController/mongo-rc2
Containers:
  mongo-node2:
    Container ID:
    Image:              mongo
    Image ID:
    Port:               27017/TCP
    Command:
      mongod
      --replSet
      bippo
      --storageEngine
      wiredTiger
      --keyFile
      /etc/mongo-conf/mongo.keyfile
    Requests:
      cpu:                      100m
    State:                      Waiting
      Reason:                   ContainerCreating
    Ready:                      False
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  mongo-conf:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongo-conf
    FSType:     ext4
    Partition:  0
    ReadOnly:   true
  mongo-persistent-storage2:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongodb-disk2
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-i0lox:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-i0lox
QoS Tier:       Burstable
Events:
  FirstSeen     LastSeen        Count   From                                            SubobjectPath   Type
Reason          Message
  ---------     --------        -----   ----                                            -------------   --------
------          -------
  30m           30m             1       {default-scheduler }                                        Normal
Scheduled       Successfully assigned mongo-rc2-aktox to gke-fatih-small-pool-59881027-h46g
  28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                Warning
FailedMount     Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
  28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                Warning
FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]"><pre class="notranslate"><code class="notranslate">kubectl describe po mongo-rc2-aktox
Name:           mongo-rc2-aktox
Namespace:      default
Node:           gke-fatih-small-pool-59881027-h46g/10.142.0.6
Start Time:     Tue, 23 Aug 2016 05:01:49 +0700
Labels:         instance=fatih2
                name=mongo-node2
Status:         Pending
IP:
Controllers:    ReplicationController/mongo-rc2
Containers:
  mongo-node2:
    Container ID:
    Image:              mongo
    Image ID:
    Port:               27017/TCP
    Command:
      mongod
      --replSet
      bippo
      --storageEngine
      wiredTiger
      --keyFile
      /etc/mongo-conf/mongo.keyfile
    Requests:
      cpu:                      100m
    State:                      Waiting
      Reason:                   ContainerCreating
    Ready:                      False
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  mongo-conf:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongo-conf
    FSType:     ext4
    Partition:  0
    ReadOnly:   true
  mongo-persistent-storage2:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mongodb-disk2
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-i0lox:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-i0lox
QoS Tier:       Burstable
Events:
  FirstSeen     LastSeen        Count   From                                            SubobjectPath   Type
Reason          Message
  ---------     --------        -----   ----                                            -------------   --------
------          -------
  30m           30m             1       {default-scheduler }                                        Normal
Scheduled       Successfully assigned mongo-rc2-aktox to gke-fatih-small-pool-59881027-h46g
  28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                Warning
FailedMount     Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
  28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                Warning
FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Relevant <code class="notranslate">/var/log/kubelet.log</code> in the node:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n100 /var/log/kubelet.log  | grep mongo-conf

E0822 22:30:30.157423    3439 kubelet.go:1932] Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:30:30.157465    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
I0822 22:31:05.117374    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:31:05.120603    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:33:05.120584643 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status.
E0822 22:32:45.156181    3439 kubelet.go:1932] Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:32:45.156226    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
I0822 22:33:05.156100    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:33:05.159285    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:35:05.159269631 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status.
E0822 22:34:58.157744    3439 kubelet.go:1932] Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:34:58.161069    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
I0822 22:35:05.188747    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:35:05.191576    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:37:05.191557765 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status."><pre class="notranslate"><code class="notranslate">ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n100 /var/log/kubelet.log  | grep mongo-conf

E0822 22:30:30.157423    3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:30:30.157465    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:31:05.117374    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:31:05.120603    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:33:05.120584643 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:32:45.156181    3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:32:45.156226    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:33:05.156100    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:33:05.159285    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:35:05.159269631 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:34:58.157744    3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:34:58.161069    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:35:05.188747    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:35:05.191576    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:37:05.191557765 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
</code></pre></div>
<p dir="auto"><code class="notranslate">/var/log/kubelet.log</code> for 50 last lines:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:22.870417    3439 server.go:959] GET /healthz: (28.36µs) 200 [[curl/7.26.0] 127.0.0.1:47687]
I0822 22:35:32.239049    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) to pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:35:32.242781    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;).
I0822 22:35:32.883593    3439 server.go:959] GET /healthz: (42.657µs) 200 [[curl/7.26.0] 127.0.0.1:47694]
I0822 22:35:38.160222    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:35:38.163717    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;).
I0822 22:35:38.537711    3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284    3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178    3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869    3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580    3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348    3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215    3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851    3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762    3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;).
I0822 22:36:39.663850    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584    3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) to pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;).
I0822 22:36:52.988636    3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414    3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193    3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:37:05.236650    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status.
E0822 22:37:09.156182    3439 kubelet.go:1932] Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933    3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881    3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;).
I0822 22:37:33.042085    3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583    3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076    3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791    3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970    3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454    3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;).
I0822 22:38:11.184431    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) to pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;).
I0822 22:38:13.101813    3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472    3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366    3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838    3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960    3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
ceefour@gke-fatih-small-pool-59881027-h46g:~$
ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:38.537711    3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284    3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178    3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869    3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580    3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348    3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215    3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851    3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762    3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;).
I0822 22:36:39.663850    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584    3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) to pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;).
I0822 22:36:52.988636    3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414    3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193    3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:37:05.236650    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status.
E0822 22:37:09.156182    3439 kubelet.go:1932] Unable to mount volumes for pod &quot;mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;mongo-rc2-aktox&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933    3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881    3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;).
I0822 22:37:33.042085    3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583    3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076    3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791    3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970    3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454    3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) to pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox&quot; (spec.Name: &quot;default-token-i0lox&quot;) pod &quot;ad21d245-689f-11e6-b129-42010af0011e&quot; (UID: &quot;ad21d245-689f-11e6-b129-42010af0011e&quot;).
I0822 22:38:11.184431    3439 reconciler.go:254] MountVolume operation started for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) to pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix&quot; (spec.Name: &quot;default-token-sb5ix&quot;) pod &quot;33f45369-687d-11e6-b129-42010af0011e&quot; (UID: &quot;33f45369-687d-11e6-b129-42010af0011e&quot;).
I0822 22:38:13.101813    3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472    3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366    3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838    3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960    3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
I0822 22:38:53.169820    3439 server.go:959] GET /healthz: (25µs) 200 [[curl/7.26.0] 127.0.0.1:47843]
I0822 22:39:03.181619    3439 server.go:959] GET /healthz: (29.069µs) 200 [[curl/7.26.0] 127.0.0.1:47849]
I0822 22:39:05.018297    3439 server.go:959] GET /stats/summary/: (6.900965ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:39:05.318114    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;)
E0822 22:39:05.321046    3439 nestedpendingoperations.go:233] Operation for &quot;\&quot;kubernetes.io/gce-pd/mongo-conf\&quot;&quot; failed. No retries permitted until 2016-08-22 22:41:05.321027505 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume &quot;kubernetes.io/gce-pd/mongo-conf&quot; (spec.Name: &quot;mongo-conf&quot;) pod &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot; (UID: &quot;06ce97cd-68b4-11e6-b129-42010af0011e&quot;) is not yet attached according to node status.
I0822 22:39:13.194669    3439 server.go:959] GET /healthz: (26.614µs) 200 [[curl/7.26.0] 127.0.0.1:47856]"><pre class="notranslate"><code class="notranslate">ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:22.870417    3439 server.go:959] GET /healthz: (28.36µs) 200 [[curl/7.26.0] 127.0.0.1:47687]
I0822 22:35:32.239049    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:35:32.242781    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:35:32.883593    3439 server.go:959] GET /healthz: (42.657µs) 200 [[curl/7.26.0] 127.0.0.1:47694]
I0822 22:35:38.160222    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:35:38.163717    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:35:38.537711    3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284    3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178    3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869    3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580    3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348    3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215    3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851    3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762    3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:36:39.663850    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584    3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:36:52.988636    3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414    3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193    3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:37:05.236650    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:37:09.156182    3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933    3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881    3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e").
I0822 22:37:33.042085    3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583    3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076    3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791    3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970    3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454    3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:38:11.184431    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:38:13.101813    3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472    3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366    3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838    3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960    3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
ceefour@gke-fatih-small-pool-59881027-h46g:~$
ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log
I0822 22:35:38.537711    3439 server.go:959] GET /healthz: (44.537µs) 200 [[Go-http-client/1.1] 127.0.0.1:47701]
I0822 22:35:39.660496    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:35:42.896284    3439 server.go:959] GET /healthz: (38.658µs) 200 [[curl/7.26.0] 127.0.0.1:47705]
I0822 22:35:52.910178    3439 server.go:959] GET /healthz: (25.065µs) 200 [[curl/7.26.0] 127.0.0.1:47712]
I0822 22:36:02.922869    3439 server.go:959] GET /healthz: (24.963µs) 200 [[curl/7.26.0] 127.0.0.1:47717]
I0822 22:36:05.013580    3439 server.go:959] GET /stats/summary/: (6.953162ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:36:12.935348    3439 server.go:959] GET /healthz: (27.682µs) 200 [[curl/7.26.0] 127.0.0.1:47724]
I0822 22:36:22.948215    3439 server.go:959] GET /healthz: (28.114µs) 200 [[curl/7.26.0] 127.0.0.1:47731]
I0822 22:36:32.961851    3439 server.go:959] GET /healthz: (44.817µs) 200 [[curl/7.26.0] 127.0.0.1:47738]
I0822 22:36:38.537762    3439 server.go:959] GET /healthz: (47.175µs) 200 [[Go-http-client/1.1] 127.0.0.1:47743]
I0822 22:36:39.176725    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:39.189075    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:36:39.663850    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:36:42.974584    3439 server.go:959] GET /healthz: (40.231µs) 200 [[curl/7.26.0] 127.0.0.1:47746]
I0822 22:36:43.184638    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:36:43.187674    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:36:52.988636    3439 server.go:959] GET /healthz: (40.953µs) 200 [[curl/7.26.0] 127.0.0.1:47754]
I0822 22:37:03.003414    3439 server.go:959] GET /healthz: (42.733µs) 200 [[curl/7.26.0] 127.0.0.1:47759]
I0822 22:37:05.034193    3439 server.go:959] GET /stats/summary/: (6.947947ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:37:05.233719    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:37:05.236650    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:39:05.23662061 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
E0822 22:37:09.156182    3439 kubelet.go:1932] Unable to mount volumes for pod "mongo-rc2-aktox_default(06ce97cd-68b4-11e6-b129-42010af0011e)": timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]; skipping pod
E0822 22:37:09.156242    3439 pod_workers.go:183] Error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod "mongo-rc2-aktox"/"default". list of unattached/unmounted volumes=[mongo-conf]
I0822 22:37:13.015933    3439 server.go:959] GET /healthz: (54.514µs) 200 [[curl/7.26.0] 127.0.0.1:47768]
I0822 22:37:23.028881    3439 server.go:959] GET /healthz: (43.187µs) 200 [[curl/7.26.0] 127.0.0.1:47775]
I0822 22:37:23.185043    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:37:23.188665    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e").
I0822 22:37:33.042085    3439 server.go:959] GET /healthz: (42.351µs) 200 [[curl/7.26.0] 127.0.0.1:47784]
I0822 22:37:38.537583    3439 server.go:959] GET /healthz: (40.577µs) 200 [[Go-http-client/1.1] 127.0.0.1:47789]
I0822 22:37:39.667127    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:37:43.055076    3439 server.go:959] GET /healthz: (26.24µs) 200 [[curl/7.26.0] 127.0.0.1:47792]
I0822 22:37:53.067791    3439 server.go:959] GET /healthz: (26.387µs) 200 [[curl/7.26.0] 127.0.0.1:47800]
I0822 22:38:03.089970    3439 server.go:959] GET /healthz: (38.445µs) 200 [[curl/7.26.0] 127.0.0.1:47805]
I0822 22:38:05.013454    3439 server.go:959] GET /stats/summary/: (7.716171ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:38:06.174617    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") to pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:06.177284    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox" (spec.Name: "default-token-i0lox") pod "ad21d245-689f-11e6-b129-42010af0011e" (UID: "ad21d245-689f-11e6-b129-42010af0011e").
I0822 22:38:11.184431    3439 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") to pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e"). Volume is already mounted to pod, but remount was requested.
I0822 22:38:11.187573    3439 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix" (spec.Name: "default-token-sb5ix") pod "33f45369-687d-11e6-b129-42010af0011e" (UID: "33f45369-687d-11e6-b129-42010af0011e").
I0822 22:38:13.101813    3439 server.go:959] GET /healthz: (35.796µs) 200 [[curl/7.26.0] 127.0.0.1:47814]
I0822 22:38:23.122472    3439 server.go:959] GET /healthz: (36.311µs) 200 [[curl/7.26.0] 127.0.0.1:47821]
I0822 22:38:33.136366    3439 server.go:959] GET /healthz: (40.312µs) 200 [[curl/7.26.0] 127.0.0.1:47828]
I0822 22:38:38.537838    3439 server.go:959] GET /healthz: (44.226µs) 200 [[Go-http-client/1.1] 127.0.0.1:47833]
I0822 22:38:39.670329    3439 container_manager_linux.go:614] Found 45 PIDs in root, 45 of them are not to be moved
I0822 22:38:43.157960    3439 server.go:959] GET /healthz: (34.945µs) 200 [[curl/7.26.0] 127.0.0.1:47836]
I0822 22:38:53.169820    3439 server.go:959] GET /healthz: (25µs) 200 [[curl/7.26.0] 127.0.0.1:47843]
I0822 22:39:03.181619    3439 server.go:959] GET /healthz: (29.069µs) 200 [[curl/7.26.0] 127.0.0.1:47849]
I0822 22:39:05.018297    3439 server.go:959] GET /stats/summary/: (6.900965ms) 200 [[Go-http-client/1.1] 10.60.2.3:57402]
I0822 22:39:05.318114    3439 reconciler.go:180] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e")
E0822 22:39:05.321046    3439 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/mongo-conf\"" failed. No retries permitted until 2016-08-22 22:41:05.321027505 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/gce-pd/mongo-conf" (spec.Name: "mongo-conf") pod "06ce97cd-68b4-11e6-b129-42010af0011e" (UID: "06ce97cd-68b4-11e6-b129-42010af0011e") is not yet attached according to node status.
I0822 22:39:13.194669    3439 server.go:959] GET /healthz: (26.614µs) 200 [[curl/7.26.0] 127.0.0.1:47856]
</code></pre></div>
<p dir="auto">Url</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="https://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes/v1.0.1/docs/api-reference/definitions.html#_v1_pod"><pre class="notranslate"><code class="notranslate">https://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes/v1.0.1/docs/api-reference/definitions.html#_v1_pod
</code></pre></div>
<p dir="auto">Documentation Page</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="http://kubernetes.io/v1.0/docs/user-guide/pods.html"><pre class="notranslate"><code class="notranslate">http://kubernetes.io/v1.0/docs/user-guide/pods.html
</code></pre></div>
<p dir="auto">Section API Object</p>