<p dir="auto">When a node is both master and minion --- that is, the <code class="notranslate">provision-masterandnode</code> function in <code class="notranslate">util.sh</code> is invoked, the flannel files for both master and minion are copied to the same node.  That looks bad, the file names are the same but the contents are different.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): unable to unmount volume properly</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: vSphere</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RETTY_NAME=&quot;Debian GNU/Linux 8 (jessie)&quot;
NAME=&quot;Debian GNU/Linux&quot;
VERSION_ID=&quot;8&quot;
VERSION=&quot;8 (jessie)&quot;
ID=debian
HOME_URL=&quot;http://www.debian.org/&quot;
SUPPORT_URL=&quot;http://www.debian.org/support/&quot;
BUG_REPORT_URL=&quot;https://bugs.debian.org/&quot;"><pre class="notranslate"><code class="notranslate">RETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="https://bugs.debian.org/"
</code></pre></div>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
<code class="notranslate">Linux kubernetes-master 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux</code></li>
<li><strong>Install tools</strong>: None</li>
<li><strong>Others</strong>: None</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
A deployment was created with the volume "<strong>kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master</strong>" mounted on the pod. Existing deployment was deleted and recreated. When I checked the logs during the delete of the deployment, I unmounts the volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" successfully. But, I also tries to unmount a volume "<strong>kubernetes.io/vsphere-volume/hello-master</strong>" which doesn't exist at all. This seems to happen only with kubernetes v1.4.5 but v1.4.4 works perfectly fine.<br>
You can check the logs below.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537562   23426 reconciler.go:166] Attempting to start UnmountVolume for volume &quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537760   23426 reconciler.go:188] UnmountVolume operation started for volume &quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537944   23426 reconciler.go:166] Attempting to start UnmountVolume for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.538142   23426 reconciler.go:188] UnmountVolume operation started for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

It loops on forever trying to unmount to the volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; which doesn't exist at all.
"><pre class="notranslate"><code class="notranslate">Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537562   23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537760   23426 reconciler.go:188] UnmountVolume operation started for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537944   23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.538142   23426 reconciler.go:188] UnmountVolume operation started for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

It loops on forever trying to unmount to the volume "kubernetes.io/vsphere-volume/hello-master" which doesn't exist at all.

</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
All the pods in the recreated deployment comes up.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
See above.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
Here are some logs. I also see multiple calls to TearDown().</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.552683   23426 nestedpendingoperations.go:253] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master\&quot; (\&quot;d470b123-aadd-11e6-bd75-005056b52a82\&quot;)&quot; failed. No retries permitted until 2016-11-14 22:00:48.052644231 -0500 EST (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume &quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master&quot; (volume.spec.Name: &quot;hello-master&quot;) pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;) with: Unmount failed: exit status 32

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Unmounting arguments: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Output: umount: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master not mounted

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.639638   23426 reconciler.go:166] Attempting to start UnmountVolume for volume &quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640356   23426 reconciler.go:320] Attempting to start UnmountDevice for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;)

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640558   23426 reconciler.go:338] UnmountDevice operation started for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;)

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642090   23426 mount.go:121] could not determine device for path: &quot;/var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master&quot;

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642277   23426 util.go:72] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.642490   23426 nestedpendingoperations.go:253] Operation for &quot;\&quot;kubernetes.io/vsphere-volume/hello-master\&quot;&quot; failed. No retries permitted until 2016-11-14 22:00:48.142466159 -0500 EST (durationBeforeRetry 500ms). Error: UnmountDevice.DeviceOpened failed for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;) with: PathIsDevice failed for path &quot;&quot;: stat : no such file or directory

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741368   23426 reconciler.go:166] Attempting to start UnmountVolume for volume &quot;kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master&quot; (spec.Name: &quot;hello-master&quot;) from pod &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot; (UID: &quot;d470b123-aadd-11e6-bd75-005056b52a82&quot;).

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741641   23426 reconciler.go:320] Attempting to start UnmountDevice for volume &quot;kubernetes.io/vsphere-volume/hello-master&quot; (spec.Name: &quot;hello-master&quot;)
&gt; "><pre class="notranslate"><code class="notranslate">&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.552683   23426 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master\" (\"d470b123-aadd-11e6-bd75-005056b52a82\")" failed. No retries permitted until 2016-11-14 22:00:48.052644231 -0500 EST (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (volume.spec.Name: "hello-master") pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82") with: Unmount failed: exit status 32

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Unmounting arguments: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Output: umount: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master not mounted

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.639638   23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640356   23426 reconciler.go:320] Attempting to start UnmountDevice for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640558   23426 reconciler.go:338] UnmountDevice operation started for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642090   23426 mount.go:121] could not determine device for path: "/var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master"

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642277   23426 util.go:72] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.642490   23426 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/vsphere-volume/hello-master\"" failed. No retries permitted until 2016-11-14 22:00:48.142466159 -0500 EST (durationBeforeRetry 500ms). Error: UnmountDevice.DeviceOpened failed for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") with: PathIsDevice failed for path "": stat : no such file or directory

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741368   23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

&gt; Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741641   23426 reconciler.go:320] Attempting to start UnmountDevice for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")
&gt; 
</code></pre></div>
<p dir="auto">Also, I see an orphan mount point on the node. Even after deleting this mount point on the node, I still see issues with the new deployment.</p>
<p dir="auto">Also we see a similar issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="187456517" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36269" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/36269/hovercard" href="https://github.com/kubernetes/kubernetes/issues/36269">#36269</a>, however cherry picking the fix from jingxu97:syncAttach-10-15 for this bug did not solve the problem for us.</p>