<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): vSphere, pod, timeout</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug Report</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2+coreos.0", GitCommit:"4fbc7c924d1e09ef018598bd053b596eb9bdd95c", GitTreeState:"clean", BuildDate:"2016-11-29T01:43:08Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: vSphere Cloud Provider - vCenter Server v6.5.0, ESXi 6.5.0</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1185.3.0 (MoreOS)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux k8-w2 4.7.3-coreos-r2</li>
<li><strong>Install tools</strong>: N/A</li>
<li><strong>Others</strong>: N/A</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I created a Pod with the following spec, using <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/abrarshivani/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/abrarshivani">@abrarshivani</a> 's example <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/vsphere#volumes">here</a> ;</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: pvpod
spec:
  containers:
  - name: test-container
    image: gcr.io/google_containers/test-webserver
    volumeMounts:
    - name: vmdk-storage
      mountPath: /test-vmdk
  volumes:
  - name: vmdk-storage
    vsphereVolume:
      volumePath: &quot;[kstore-k8s-vol] volumes/test&quot;
      fsType: ext4"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: pvpod
spec:
  containers:
  - name: test-container
    image: gcr.io/google_containers/test-webserver
    volumeMounts:
    - name: vmdk-storage
      mountPath: /test-vmdk
  volumes:
  - name: vmdk-storage
    vsphereVolume:
      volumePath: "[kstore-k8s-vol] volumes/test"
      fsType: ext4
</code></pre></div>
<p dir="auto">The VMDK specified in the volumePath was successfully attached to the VM that the pod was scheduled in, but the pod becomes stuck in the containerCreating state. After 2 minutes, the kubelet on the worker node that the pod was scheduled on logs a timeout error.</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
I expected the Pod to be created with the VMDK mounted under /test-vmdk inside the pod.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Create a new VMDK using the command <code class="notranslate">vmkfstools -c 2G /vmfs/volumes/kstore-k8s-vol/volumes/test.vmdk</code></li>
<li>Create a new Pod using the following spec;</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: pvpod
spec:
  containers:
  - name: test-container
    image: gcr.io/google_containers/test-webserver
    volumeMounts:
    - name: vmdk-storage
      mountPath: /test-vmdk
  volumes:
  - name: vmdk-storage
    vsphereVolume:
      volumePath: &quot;[kstore-k8s-vol] volumes/test&quot;
      fsType: ext4"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: pvpod
spec:
  containers:
  - name: test-container
    image: gcr.io/google_containers/test-webserver
    volumeMounts:
    - name: vmdk-storage
      mountPath: /test-vmdk
  volumes:
  - name: vmdk-storage
    vsphereVolume:
      volumePath: "[kstore-k8s-vol] volumes/test"
      fsType: ext4
</code></pre></div>
<ol start="3" dir="auto">
<li>Observe errors by running <code class="notranslate">kubectl describe pod pvpod</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                    -------------   --------        ------          -------
  13m           13m             1       {default-scheduler }                    Normal          Scheduled       Successfully assigned pvpod to k8-w2
  11m           17s             6       {kubelet k8-w2}                         Warning         FailedMount     Unable to mount volumes for pod &quot;pvpod_default(ec696999-ba3d-11e6-a9a4-005056892975)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;pvpod&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[vmdk-storage]
  11m           17s             6       {kubelet k8-w2}                         Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;pvpod&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[vmdk-storage]"><pre class="notranslate"><code class="notranslate">  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                    -------------   --------        ------          -------
  13m           13m             1       {default-scheduler }                    Normal          Scheduled       Successfully assigned pvpod to k8-w2
  11m           17s             6       {kubelet k8-w2}                         Warning         FailedMount     Unable to mount volumes for pod "pvpod_default(ec696999-ba3d-11e6-a9a4-005056892975)": timeout expired waiting for volumes to attach/mount for pod "pvpod"/"default". list of unattached/unmounted volumes=[vmdk-storage]
  11m           17s             6       {kubelet k8-w2}                         Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "pvpod"/"default". list of unattached/unmounted volumes=[vmdk-storage]
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">kube-controller-manager log entries;</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-12-04T16:23:00.422463105Z I1204 16:23:00.422244       1 reconciler.go:202] Started AttachVolume for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; to node &quot;k8-w2&quot;
2016-12-04T16:23:00.429345287Z W1204 16:23:00.429078       1 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
2016-12-04T16:23:01.491332560Z I1204 16:23:01.491246       1 operation_executor.go:619] AttachVolume.Attach succeeded for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) from node &quot;k8-w2&quot;."><pre class="notranslate"><code class="notranslate">2016-12-04T16:23:00.422463105Z I1204 16:23:00.422244       1 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" to node "k8-w2"
2016-12-04T16:23:00.429345287Z W1204 16:23:00.429078       1 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
2016-12-04T16:23:01.491332560Z I1204 16:23:01.491246       1 operation_executor.go:619] AttachVolume.Attach succeeded for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") from node "k8-w2".
</code></pre></div>
<p dir="auto">k8-w2 worker node kubelet logs;</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: E1204 16:23:00.488905     946 nestedpendingoperations.go:262] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\&quot;&quot; failed. No retries permitted until 2016-12-04 16:23:00.988885317 +0000 UTC (durationBeforeRetry 500ms). Error: Volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.488827     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.490608     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt&quot; (spec.Name: &quot;default-token-zcpmt&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.608354     946 operation_executor.go:916] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt&quot; (spec.Name: &quot;default-token-zcpmt&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;).
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.991910     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: E1204 16:23:00.992429     946 nestedpendingoperations.go:262] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\&quot;&quot; failed. No retries permitted until 2016-12-04 16:23:01.992407088 +0000 UTC (durationBeforeRetry 1s). Error: Volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:02 k8-w2 kubelet-wrapper[946]: I1204 16:23:02.000818     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:02 k8-w2 kubelet-wrapper[946]: E1204 16:23:02.001589     946 nestedpendingoperations.go:262] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\&quot;&quot; failed. No retries permitted until 2016-12-04 16:23:04.00155815 +0000 UTC (durationBeforeRetry 2s). Error: Volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:04 k8-w2 kubelet-wrapper[946]: I1204 16:23:04.008834     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:04 k8-w2 kubelet-wrapper[946]: E1204 16:23:04.009560     946 nestedpendingoperations.go:262] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\&quot;&quot; failed. No retries permitted until 2016-12-04 16:23:08.00952202 +0000 UTC (durationBeforeRetry 4s). Error: Volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:05 k8-w2 kubelet-wrapper[946]: W1204 16:23:05.154304     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.035413     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;)
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.040480     946 operation_executor.go:1199] Controller successfully attached volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) devicePath: &quot;/dev/disk/by-id/wwn-0x6000c29644981da81416d23b00969968&quot;
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.136359     946 operation_executor.go:811] Entering MountVolume.WaitForAttach for volume &quot;kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test&quot; (spec.Name: &quot;vmdk-storage&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;) DevicePath: &quot;/dev/disk/by-id/wwn-0x6000c29644981da81416d23b00969968&quot;
Dec 04 16:23:15 k8-w2 kubelet-wrapper[946]: W1204 16:23:15.708296     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:26 k8-w2 kubelet-wrapper[946]: W1204 16:23:26.292271     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:36 k8-w2 kubelet-wrapper[946]: W1204 16:23:36.833943     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:47 k8-w2 kubelet-wrapper[946]: W1204 16:23:47.284526     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:57 k8-w2 kubelet-wrapper[946]: W1204 16:23:57.759809     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:08 k8-w2 kubelet-wrapper[946]: W1204 16:24:08.260559     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:18 k8-w2 kubelet-wrapper[946]: W1204 16:24:18.740360     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:29 k8-w2 kubelet-wrapper[946]: W1204 16:24:29.271524     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:39 k8-w2 kubelet-wrapper[946]: W1204 16:24:39.857839     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:50 k8-w2 kubelet-wrapper[946]: W1204 16:24:50.560441     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:00 k8-w2 kubelet-wrapper[946]: E1204 16:25:00.361456     946 kubelet.go:1521] Unable to mount volumes for pod &quot;pvpod_default(ec696999-ba3d-11e6-a9a4-005056892975)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;pvpod&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[vmdk-storage]; skipping pod
Dec 04 16:25:00 k8-w2 kubelet-wrapper[946]: E1204 16:25:00.362208     946 pod_workers.go:184] Error syncing pod ec696999-ba3d-11e6-a9a4-005056892975, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;pvpod&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[vmdk-storage]
Dec 04 16:25:01 k8-w2 kubelet-wrapper[946]: W1204 16:25:01.037741     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:11 k8-w2 kubelet-wrapper[946]: W1204 16:25:11.531951     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:12 k8-w2 kubelet-wrapper[946]: I1204 16:25:12.847941     946 operation_executor.go:916] MountVolume.SetUp succeeded for volume &quot;kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt&quot; (spec.Name: &quot;default-token-zcpmt&quot;) pod &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot; (UID: &quot;ec696999-ba3d-11e6-a9a4-005056892975&quot;).
Dec 04 16:25:22 k8-w2 kubelet-wrapper[946]: W1204 16:25:22.065628     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:32 k8-w2 kubelet-wrapper[946]: W1204 16:25:32.635166     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:43 k8-w2 kubelet-wrapper[946]: W1204 16:25:43.113635     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:53 k8-w2 kubelet-wrapper[946]: W1204 16:25:53.599072     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:04 k8-w2 kubelet-wrapper[946]: W1204 16:26:04.074564     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:14 k8-w2 kubelet-wrapper[946]: W1204 16:26:14.750181     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:25 k8-w2 kubelet-wrapper[946]: W1204 16:26:25.399264     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:35 k8-w2 kubelet-wrapper[946]: W1204 16:26:35.875540     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:46 k8-w2 kubelet-wrapper[946]: W1204 16:26:46.364960     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated"><pre class="notranslate"><code class="notranslate">Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: E1204 16:23:00.488905     946 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\"" failed. No retries permitted until 2016-12-04 16:23:00.988885317 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.488827     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.490608     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt" (spec.Name: "default-token-zcpmt") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.608354     946 operation_executor.go:916] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt" (spec.Name: "default-token-zcpmt") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975").
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: I1204 16:23:00.991910     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:00 k8-w2 kubelet-wrapper[946]: E1204 16:23:00.992429     946 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\"" failed. No retries permitted until 2016-12-04 16:23:01.992407088 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:02 k8-w2 kubelet-wrapper[946]: I1204 16:23:02.000818     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:02 k8-w2 kubelet-wrapper[946]: E1204 16:23:02.001589     946 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\"" failed. No retries permitted until 2016-12-04 16:23:04.00155815 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:04 k8-w2 kubelet-wrapper[946]: I1204 16:23:04.008834     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:04 k8-w2 kubelet-wrapper[946]: E1204 16:23:04.009560     946 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test\"" failed. No retries permitted until 2016-12-04 16:23:08.00952202 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") has not yet been added to the list of VolumesInUse in the node's volume status.
Dec 04 16:23:05 k8-w2 kubelet-wrapper[946]: W1204 16:23:05.154304     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.035413     946 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975")
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.040480     946 operation_executor.go:1199] Controller successfully attached volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") devicePath: "/dev/disk/by-id/wwn-0x6000c29644981da81416d23b00969968"
Dec 04 16:23:08 k8-w2 kubelet-wrapper[946]: I1204 16:23:08.136359     946 operation_executor.go:811] Entering MountVolume.WaitForAttach for volume "kubernetes.io/vsphere-volume/[kstore-k8s-vol] volumes/test" (spec.Name: "vmdk-storage") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975") DevicePath: "/dev/disk/by-id/wwn-0x6000c29644981da81416d23b00969968"
Dec 04 16:23:15 k8-w2 kubelet-wrapper[946]: W1204 16:23:15.708296     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:26 k8-w2 kubelet-wrapper[946]: W1204 16:23:26.292271     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:36 k8-w2 kubelet-wrapper[946]: W1204 16:23:36.833943     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:47 k8-w2 kubelet-wrapper[946]: W1204 16:23:47.284526     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:23:57 k8-w2 kubelet-wrapper[946]: W1204 16:23:57.759809     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:08 k8-w2 kubelet-wrapper[946]: W1204 16:24:08.260559     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:18 k8-w2 kubelet-wrapper[946]: W1204 16:24:18.740360     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:29 k8-w2 kubelet-wrapper[946]: W1204 16:24:29.271524     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:39 k8-w2 kubelet-wrapper[946]: W1204 16:24:39.857839     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:24:50 k8-w2 kubelet-wrapper[946]: W1204 16:24:50.560441     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:00 k8-w2 kubelet-wrapper[946]: E1204 16:25:00.361456     946 kubelet.go:1521] Unable to mount volumes for pod "pvpod_default(ec696999-ba3d-11e6-a9a4-005056892975)": timeout expired waiting for volumes to attach/mount for pod "pvpod"/"default". list of unattached/unmounted volumes=[vmdk-storage]; skipping pod
Dec 04 16:25:00 k8-w2 kubelet-wrapper[946]: E1204 16:25:00.362208     946 pod_workers.go:184] Error syncing pod ec696999-ba3d-11e6-a9a4-005056892975, skipping: timeout expired waiting for volumes to attach/mount for pod "pvpod"/"default". list of unattached/unmounted volumes=[vmdk-storage]
Dec 04 16:25:01 k8-w2 kubelet-wrapper[946]: W1204 16:25:01.037741     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:11 k8-w2 kubelet-wrapper[946]: W1204 16:25:11.531951     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:12 k8-w2 kubelet-wrapper[946]: I1204 16:25:12.847941     946 operation_executor.go:916] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/ec696999-ba3d-11e6-a9a4-005056892975-default-token-zcpmt" (spec.Name: "default-token-zcpmt") pod "ec696999-ba3d-11e6-a9a4-005056892975" (UID: "ec696999-ba3d-11e6-a9a4-005056892975").
Dec 04 16:25:22 k8-w2 kubelet-wrapper[946]: W1204 16:25:22.065628     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:32 k8-w2 kubelet-wrapper[946]: W1204 16:25:32.635166     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:43 k8-w2 kubelet-wrapper[946]: W1204 16:25:43.113635     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:25:53 k8-w2 kubelet-wrapper[946]: W1204 16:25:53.599072     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:04 k8-w2 kubelet-wrapper[946]: W1204 16:26:04.074564     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:14 k8-w2 kubelet-wrapper[946]: W1204 16:26:14.750181     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:25 k8-w2 kubelet-wrapper[946]: W1204 16:26:25.399264     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:35 k8-w2 kubelet-wrapper[946]: W1204 16:26:35.875540     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
Dec 04 16:26:46 k8-w2 kubelet-wrapper[946]: W1204 16:26:46.364960     946 vsphere.go:368] Creating new client session since the existing session is not valid or not authenticated
</code></pre></div>
<p dir="auto">VM hardware with VMDK showing as being attached;</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6966796/20867559/a8b8411c-ba3f-11e6-95e8-1798346395b3.png"><img src="https://cloud.githubusercontent.com/assets/6966796/20867559/a8b8411c-ba3f-11e6-95e8-1798346395b3.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): logging, logging level</p>
<hr>
<p dir="auto">I am using <code class="notranslate">--logtostderr=true</code> and I would like to be able to see only WARN/ERROR level messages in my log of Kubernetes components.</p>
<p dir="auto">I am especially interested in being able to dismiss INFO level messages from kubelet.</p>
<p dir="auto">According to this: <a href="http://rpg.ifi.uzh.ch/docs/glog.html" rel="nofollow">http://rpg.ifi.uzh.ch/docs/glog.html</a>, it should be possible to specify minimum log level with a <code class="notranslate">minloglevel</code> flag which is not exposed on kubelet. <code class="notranslate">--stderrthreshold=2</code> didn't work -- I still saw a lot of INFO messages in my logs.</p>