<p dir="auto">hello, when i downloaded "script auto setup" and ran it to create cluster on AWS, and got this error:<br>
./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 518: KUBE_MANIFESTS_TAR_URL: unbound variable</p>
<p dir="auto">I don't see this on older version, and don't know variable "KUBE_MANIFESTS_TAR_URL", does anyone explain this ?.</p>
<p dir="auto">Thanks so much!!</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<ul dir="auto">
<li>No</li>
</ul>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>ebs</li>
<li>ebs detach</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2&quot;, GitCommit:&quot;0776eab45fe28f02bbeac0f05ae1a203051a21eb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-24T22:35:03Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2&quot;, GitCommit:&quot;0776eab45fe28f02bbeac0f05ae1a203051a21eb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-24T22:30:23Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Linux kubecontroller-1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux"><pre class="notranslate"><code class="notranslate">Linux kubecontroller-1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
</code></pre></div>
<ul dir="auto">
<li><strong>Install tools</strong>: None</li>
<li><strong>Others</strong>: None</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">I've seen this behavior in 1.4.6 and earlier.  When a pod with a <code class="notranslate">PersistentVolumeClaim</code>, in this case when using a <code class="notranslate">StorageClass</code>, is "moved" to another node (terminated on one node then run on the other), the persistent volume occasionally has trouble detaching to the old node. The move can be either from <code class="notranslate">kubectl apply -f &lt;new_configuration.yml&gt;</code> or in this particular example, when draining a node, with a command such as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl drain worker-3"><pre class="notranslate"><code class="notranslate">kubectl drain worker-3
</code></pre></div>
<p dir="auto">This does sometimes work.  Even when it works, there are plenty of errors with the EBS volume. The kube-controller will throw lots of errors, saying it can't detach.  Such as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 01 17:41:47 kubecontroller-1 kube-controller-manager[5662]: E1201 17:41:47.344806    5662 attacher.go:73] Error attaching volume &quot;aws://us-east-1c/vol-802faf11&quot;: Error attaching EBS volume &quot;vol-802faf11&quot; to instance &quot;i-ca043f59&quot;: VolumeInUse: vol-802faf11 is already attached to an instance
Dec 01 17:41:47 kubecontroller-1 kube-controller-manager[5662]:         status code: 400, request id:"><pre class="notranslate"><code class="notranslate">Dec 01 17:41:47 kubecontroller-1 kube-controller-manager[5662]: E1201 17:41:47.344806    5662 attacher.go:73] Error attaching volume "aws://us-east-1c/vol-802faf11": Error attaching EBS volume "vol-802faf11" to instance "i-ca043f59": VolumeInUse: vol-802faf11 is already attached to an instance
Dec 01 17:41:47 kubecontroller-1 kube-controller-manager[5662]:         status code: 400, request id:
</code></pre></div>
<p dir="auto">There will be other <code class="notranslate">nestedpendingoperations.go</code> operation errors.  I'm not sure if that's a symptom of a misconfiguration.  Again, it appears to SOMETIMES work. IF/WHEN it works, I'll see this in the kube-controller log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:30.821032    5662 aws.go:1492] AttachVolume volume=&quot;vol-802faf11&quot; instance=&quot;i-ca043f59&quot; request returned {
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   AttachTime: 2016-12-01 17:42:30.662 +0000 UTC,
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   Device: &quot;/dev/xvdba&quot;,
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   InstanceId: &quot;i-ca043f59&quot;,
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   State: &quot;attaching&quot;,
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   VolumeId: &quot;vol-802faf11&quot;
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: }
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:30.920339    5662 aws.go:1366] Waiting for volume &quot;vol-802faf11&quot; state: actual=attaching, desired=attached
Dec 01 17:42:41 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:41.019194    5662 aws.go:1265] Releasing in-process attachment entry: ba -&gt; volume vol-802faf11"><pre class="notranslate"><code class="notranslate">Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:30.821032    5662 aws.go:1492] AttachVolume volume="vol-802faf11" instance="i-ca043f59" request returned {
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   AttachTime: 2016-12-01 17:42:30.662 +0000 UTC,
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   Device: "/dev/xvdba",
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   InstanceId: "i-ca043f59",
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   State: "attaching",
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]:   VolumeId: "vol-802faf11"
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: }
Dec 01 17:42:30 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:30.920339    5662 aws.go:1366] Waiting for volume "vol-802faf11" state: actual=attaching, desired=attached
Dec 01 17:42:41 kubecontroller-1 kube-controller-manager[5662]: I1201 17:42:41.019194    5662 aws.go:1265] Releasing in-process attachment entry: ba -&gt; volume vol-802faf11
</code></pre></div>
<p dir="auto">But it will occasionally fail and it will never detach.  On the kubelet server, this log appears:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 01 19:38:55 worker-3 kubelet[1032]: I1201 19:38:55.432000    1032 reconciler.go:189] UnmountVolume operation started for volume &quot;kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11&quot; (spec.Name: &quot;ebstest-volume&quot;) from pod &quot;fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336&quot; (UID: &quot;fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336&quot;).
Dec 01 19:38:55 worker-3 kubelet[1032]: I1201 19:38:55.432070    1032 aws_ebs.go:398] Error checking if mountpoint /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory
Dec 01 19:38:55 worker-3 kubelet[1032]: E1201 19:38:55.432132    1032 nestedpendingoperations.go:262] Operation for &quot;\&quot;kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11\&quot; (\&quot;fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\&quot;)&quot; failed. No retries permitted until 2016-12-01 19:40:55.432095954 +0000 UTC (durationBeforeRetry 2m0s). Error: UnmountVolume.TearDown failed for volume &quot;kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11&quot; (volume.spec.Name: &quot;ebstest-volume&quot;) pod &quot;fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336&quot; (UID: &quot;fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336&quot;) with: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory"><pre class="notranslate"><code class="notranslate">Dec 01 19:38:55 worker-3 kubelet[1032]: I1201 19:38:55.432000    1032 reconciler.go:189] UnmountVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11" (spec.Name: "ebstest-volume") from pod "fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336" (UID: "fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336").
Dec 01 19:38:55 worker-3 kubelet[1032]: I1201 19:38:55.432070    1032 aws_ebs.go:398] Error checking if mountpoint /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory
Dec 01 19:38:55 worker-3 kubelet[1032]: E1201 19:38:55.432132    1032 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11\" (\"fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\")" failed. No retries permitted until 2016-12-01 19:40:55.432095954 +0000 UTC (durationBeforeRetry 2m0s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11" (volume.spec.Name: "ebstest-volume") pod "fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336" (UID: "fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336") with: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory
</code></pre></div>
<p dir="auto">And it will repeat.  If I try to look in the path specified above on the particular kubelet, <code class="notranslate">/var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/</code> exists, but the <code class="notranslate">pvc-*</code> does not.</p>
<p dir="auto">On the kubelet I can actually see the mounted volume. Again, sometimes it is successful and the pod moves to another node.  sometimes not.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Upon <code class="notranslate">kubectl drain worker-3</code>, the following actions should occur:</p>
<ul dir="auto">
<li>pod terminated.</li>
<li>EBS persistent volume unmounted.</li>
<li>EBS persistent volume mounted on another node.</li>
<li>pod created and successfully run on another node where the EBS volume is.</li>
</ul>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ul dir="auto">
<li>Create a storage class for an EBS volume, persistent volume claim, and deployment that has a pod that uses the EBS volume.</li>
<li>Drain the particular node. Sometimes this will work as intended.</li>
</ul>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<ul dir="auto">
<li>I can provide the yml files as necessary.</li>
<li>There is a second EBS volume, mounted at <code class="notranslate">/dev/xvde</code>, mounted at <code class="notranslate">/mnt/ebs</code>.  This is because the standard AMI root drive is very small.  The Docker path is there and the kubelet is symlinked there <code class="notranslate">/var/lib/kubelet -&gt; /mnt/ebs/kubelet</code>. I will test this deployment WITHOUT the EBS volume and symlinks to verify.</li>
</ul>