<p dir="auto"><a href="https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/29986/kubernetes-pull-build-test-e2e-gce/51596/" rel="nofollow">https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/29986/kubernetes-pull-build-test-e2e-gce/51596/</a></p>
<p dir="auto">Failed: [k8s.io] Docker Containers should be able to override the image's default commmand (docker entrypoint) [Conformance] {Kubernetes e2e suite}</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/docker_containers.go:54
Error creating Pod
Expected error:
    &lt;*url.Error | 0xc82040c210&gt;: {
        Op: &quot;Post&quot;,
        URL: &quot;https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods&quot;,
        Err: {
            Op: &quot;dial&quot;,
            Net: &quot;tcp&quot;,
            Source: nil,
            Addr: {
                IP: &quot;\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xffh\x9b\xb1\x1c&quot;,
                Port: 443,
                Zone: &quot;&quot;,
            },
            Err: {},
        },
    }
    Post https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods: dial tcp 104.155.177.28:443: i/o timeout
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50"><pre class="notranslate"><code class="notranslate">/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/docker_containers.go:54
Error creating Pod
Expected error:
    &lt;*url.Error | 0xc82040c210&gt;: {
        Op: "Post",
        URL: "https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods",
        Err: {
            Op: "dial",
            Net: "tcp",
            Source: nil,
            Addr: {
                IP: "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xffh\x9b\xb1\x1c",
                Port: 443,
                Zone: "",
            },
            Err: {},
        },
    }
    Post https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods: dial tcp 104.155.177.28:443: i/o timeout
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50
</code></pre></div>
<p dir="auto">Happened on a presubmit run in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="169114911" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29986" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/29986/hovercard" href="https://github.com/kubernetes/kubernetes/pull/29986">#29986</a>.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): Completed Status POD Unmount</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5+$Format:%h$", GitCommit:"dc1efecced758680dc5aa29dc317e366b6792a48", GitTreeState:"dirty", BuildDate:"2016-10-31T10:12:09Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5+$Format:%h$", GitCommit:"996dae74127a3e5a2c3da136dd846356e612b18f", GitTreeState:"dirty", BuildDate:"2016-11-01T03:03:10Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:<br>
VMs on softlayer. 1 Master nodes 3 worker nodes</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
NAME="Ubuntu"<br>
VERSION="14.04.5 LTS, Trusty Tahr"<br>
ID=ubuntu<br>
ID_LIKE=debian<br>
PRETTY_NAME="Ubuntu 14.04.5 LTS"<br>
VERSION_ID="14.04"<br>
HOME_URL="<a href="http://www.ubuntu.com/" rel="nofollow">http://www.ubuntu.com/</a>"<br>
SUPPORT_URL="<a href="http://help.ubuntu.com/" rel="nofollow">http://help.ubuntu.com/</a>"<br>
BUG_REPORT_URL="<a href="http://bugs.launchpad.net/ubuntu/" rel="nofollow">http://bugs.launchpad.net/ubuntu/</a>"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
Linux cdsdev-dal09-stony03-api-01.softlayer.com 3.16.0-76-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35699037" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/98" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/98/hovercard" href="https://github.com/kubernetes/kubernetes/pull/98">#98</a>~14.04.1-Ubuntu SMP Fri Jun 24 17:04:54 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: Ansible</li>
<li><strong>Others</strong>: No.</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ol dir="auto">
<li>Create a bunch of PVs using nfs</li>
<li>Create PVC in a pvc_storage.yaml</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-local2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-data2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-local3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-data3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti"><pre class="notranslate"><code class="notranslate">  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-local2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-data2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-local3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dashmpp-data3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
</code></pre></div>
<ol start="3" dir="auto">
<li>After use <code class="notranslate">kubectl create -f pvc_storage.yaml</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM                     REASON    AGE
ppvl01    15Gi       RWX           Recycle         Available                                       6d
ppvl02    1Ti        RWO           Recycle         Bound       default/dashmpp-local1              6d
ppvl04    1Ti        RWO           Recycle         Bound       default/dashmpp-data1               6d
ppvl05    1Ti        RWO           Recycle         Bound       default/dashmpp-data2               27s
ppvl06    1Ti        RWO           Recycle         Bound       default/dashmpp-local2              1m
spv1      10Gi       RWX           Recycle         Bound       default/dashmpp-head                6d
spv2      10Gi       RWX           Recycle         Available                                       6d
spv3      10Gi       RWX           Recycle         Bound       default/dashmpp-scratch             6d
spv4      10Gi       RWX           Recycle         Available                                       6d"><pre class="notranslate"><code class="notranslate">NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM                     REASON    AGE
ppvl01    15Gi       RWX           Recycle         Available                                       6d
ppvl02    1Ti        RWO           Recycle         Bound       default/dashmpp-local1              6d
ppvl04    1Ti        RWO           Recycle         Bound       default/dashmpp-data1               6d
ppvl05    1Ti        RWO           Recycle         Bound       default/dashmpp-data2               27s
ppvl06    1Ti        RWO           Recycle         Bound       default/dashmpp-local2              1m
spv1      10Gi       RWX           Recycle         Bound       default/dashmpp-head                6d
spv2      10Gi       RWX           Recycle         Available                                       6d
spv3      10Gi       RWX           Recycle         Bound       default/dashmpp-scratch             6d
spv4      10Gi       RWX           Recycle         Available                                       6d
</code></pre></div>
<ol start="4" dir="auto">
<li>After run <code class="notranslate">kubectl delete -f pv_storage.yaml</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="persistentvolumeclaim &quot;dashmpp-head&quot; deleted
persistentvolumeclaim &quot;dashmpp-scratch&quot; deleted
persistentvolumeclaim &quot;dashmpp-local1&quot; deleted
persistentvolumeclaim &quot;dashmpp-data1&quot; deleted
persistentvolumeclaim &quot;dashmpp-local2&quot; deleted
persistentvolumeclaim &quot;dashmpp-data2&quot; deleted
persistentvolumeclaim &quot;dashmpp-local3&quot; deleted
persistentvolumeclaim &quot;dashmpp-data3&quot; deleted"><pre class="notranslate"><code class="notranslate">persistentvolumeclaim "dashmpp-head" deleted
persistentvolumeclaim "dashmpp-scratch" deleted
persistentvolumeclaim "dashmpp-local1" deleted
persistentvolumeclaim "dashmpp-data1" deleted
persistentvolumeclaim "dashmpp-local2" deleted
persistentvolumeclaim "dashmpp-data2" deleted
persistentvolumeclaim "dashmpp-local3" deleted
persistentvolumeclaim "dashmpp-data3" deleted
</code></pre></div>
<ol start="5" dir="auto">
<li>Using <code class="notranslate"> kubectl get pod -a -o wide</code>  some of the recyclers are in <code class="notranslate">Completed</code> status.</li>
</ol>
<p dir="auto">NAME                  READY     STATUS      RESTARTS   AGE       IP        NODE<br>
recycler-for-ppvl04   0/1       Completed   0          1m            10.121.55.231<br>
recycler-for-spv1     0/1       Completed   0          1m            10.121.55.102<br>
recycler-for-spv3     0/1       Completed   0          1m            10.121.55.70</p>
<ol start="6" dir="auto">
<li>Login one of the recycler Node, e.g. <code class="notranslate">10.121.55.70</code> using <code class="notranslate">df -h</code>, we got</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="169.55.11.79:/gpfs/fs01/shared/prod/democluster01/dashdb/gamestop/spv3   99T   18T   81T  19% /var/lib/kubelet/pods/1e5c2676-a045-11e6-8264-06d7ef65f0fb/volumes/kubernetes.io~nfs/ppvl02"><pre class="notranslate"><code class="notranslate">169.55.11.79:/gpfs/fs01/shared/prod/democluster01/dashdb/gamestop/spv3   99T   18T   81T  19% /var/lib/kubelet/pods/1e5c2676-a045-11e6-8264-06d7ef65f0fb/volumes/kubernetes.io~nfs/ppvl02
</code></pre></div>
<p dir="auto">As you can see the mount point still exist.</p>
<ol start="7" dir="auto">
<li>If we delelet PVCs one by one, such as</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl delete pvc dashmpp-local1
kubectl delete pvc dashmpp-local2
kubectl delete pvc dashmpp-data1 
kubectl delete pvc dashmpp-data2
kubectl delete pvc  dashmpp-head  
kubectl delete pvc  dashmpp-scratch   "><pre class="notranslate"><code class="notranslate">kubectl delete pvc dashmpp-local1
kubectl delete pvc dashmpp-local2
kubectl delete pvc dashmpp-data1 
kubectl delete pvc dashmpp-data2
kubectl delete pvc  dashmpp-head  
kubectl delete pvc  dashmpp-scratch   
</code></pre></div>
<p dir="auto">We do not get such problems</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Using <code class="notranslate">kubectl delete -f pvc_storage.yaml</code>, all the recycler could not in<code class="notranslate">Completed</code> status, and umount the mount point from worker node.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Describe in what happened section</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
No.</p>