<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): NO</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): various combinations of EBS/volume/deleted/wiped/1.3</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.3.5</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 899.1.0</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): <code class="notranslate">Linux ip-10-20-8-136.ec2.internal 4.3.3-coreos #2 SMP Thu Dec 17 23:57:55 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux</code></li>
<li><strong>Install tools</strong>: CloudFormation + Ansible</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">After the upgrade to 1.3.5, EBS volumes no longer get detached after their parent pods move to different nodes. Worse still, the volumes get wiped clean.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">EBS volumes should get detached/attached as pods move and their content should never be wiped.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Starting from a cluster running 1.2.4</p>
<ol dir="auto">
<li>Create a deployment of size 1 with an EBS volume, and write some data to the volume</li>
<li>Upgrade the cluster to 1.3.5. Following the instructions at <a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md/#v130-beta1">https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md/#v130-beta1</a>, under the bullet point "In order to safely upgrade an existing Kubernetes cluster without interruption of volume attach/detach logic:"
<ol dir="auto">
<li>Upgrade master to 1.3.5</li>
<li>Upgrade nodes to 1.3.5</li>
</ol>
</li>
<li>Trigger a move for the pod created in step 1, for example by specifying a different node in spec.nodeName</li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">First, I think volume attach/detach stopped working after the upgrade because kubelet can't add the <code class="notranslate">volumes.kubernetes.io/controller-managed-attach-detach</code> annotation to <em>existing</em> nodes. New nodes do come up with this annotation.</p>
<p dir="auto">Second, the volumes get wiped because kubelet tries to recursively delete the pod directory <em>before</em> the volumes are unmounted, which, because of the first problem, never happens.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thomaso-mirodin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thomaso-mirodin">@thomaso-mirodin</a></p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<h2 dir="auto">skydns, dns, docs</h2>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT -- Documentation</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">master branch</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
</ul>
<p dir="auto">None, local setup via vagrant</p>
<ul dir="auto">
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<p dir="auto">CoreOS 1122.0.233</p>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<p dir="auto">Linux c40 4.7.0-coreos-r1</p>
<ul dir="auto">
<li><strong>Install tools</strong>:</li>
</ul>
<p dir="auto">Vagrant + Ansible running on a container that provisions each host to a given state (master or minion)</p>
<p dir="auto">Kubernetes running using hyperkube in a container</p>
<ul dir="auto">
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">There is no way to run SkyDNS that I can find in the docs to specify an IP address or host of the API server that would work if the API server goes away and it started up on another host.</p>
<p dir="auto">The documentation has a couple issues.</p>
<p dir="auto">The broken link in <a href="https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns">https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns</a>:</p>
<p dir="auto">"See the example files (ReplicationController and Service),"</p>
<p dir="auto">The hyperlink to "ReplicationController" is broken and goes to <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addns/dns/skydns-rc.yaml.in">https://github.com/kubernetes/kubernetes/blob/master/cluster/addns/dns/skydns-rc.yaml.in</a></p>
<p dir="auto">And should go to ("addons" instead of "addns").</p>
<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.in">https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.in</a></p>
<p dir="auto">I tried to fix this in the source but can't find where "addns" is coming from in the markdown conversion.</p>
<p dir="auto">Secondly, in the section "How Does it work", starting with "SkyDNS depends on etcd for what to serv", the paragraph is lined out.</p>
<p dir="auto">I now understand after looking at the 1.4 branch that DNS has been completely reworked to not use kube2sky, however, this is confusing if someone is just using the master branch.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">view <a href="https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns">https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns</a></p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">master branch.</p>