<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): None</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl version                                                                          
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2.2+f64c9f2d999ceb&quot;, GitCommit:&quot;f64c9f2d999ceb157d5672e9bba6639a4c456f6e&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-29T15:21:56Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2.2+f64c9f2d999ceb&quot;, GitCommit:&quot;f64c9f2d999ceb157d5672e9bba6639a4c456f6e&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-29T15:13:51Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate">$ <span class="pl-s1">kubectl version                                                                          </span>
<span class="pl-c1">Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2.2+f64c9f2d999ceb", GitCommit:"f64c9f2d999ceb157d5672e9bba6639a4c456f6e", GitTreeState:"clean", BuildDate:"2016-11-29T15:21:56Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}</span>
<span class="pl-c1">Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2.2+f64c9f2d999ceb", GitCommit:"f64c9f2d999ceb157d5672e9bba6639a4c456f6e", GitTreeState:"clean", BuildDate:"2016-11-29T15:13:51Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}</span></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GKE</li>
</ul>
<p dir="auto"><strong>What happened</strong>: Upgrades to version 1.5 (from any previous version) change existing addons from ReplicationControllers to Deployments without deleting old ReplicationControllers. This leads to multiple versions of the addons running at the same time. There also seems to be multiple deployments of heapster as well.</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl get rc --namespace=kube-system                                                   
NAME                          DESIRED   CURRENT   READY     AGE
kube-dns-v17.1                2         2         2         1h
kubernetes-dashboard-v1.1.1   1         1         1         1h
l7-default-backend-v1.0       1         1         1         1h

$ kubectl get deployment --namespace=kube-system                                           
NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
heapster-v1.1.0        1         1         1            1           1h
heapster-v1.2.0        1         1         1            1           1h
kube-dns               1         1         1            1           1h
kubernetes-dashboard   1         1         1            1           1h
l7-default-backend     1         1         1            1           1h

$ kubectl get pods --namespace=kube-system                                                 
NAME                                                               READY     STATUS    RESTARTS   AGE
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt   1/1       Unknown   0          1h
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-mgst   1/1       Running   0          1h
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0   1/1       Running   0          1h
heapster-v1.1.0-2096339923-39key                                   2/2       Running   0          1h
heapster-v1.2.0-2168613315-1bcy3                                   2/2       Running   0          1h
kube-dns-4101612645-78hx6                                          4/4       Running   0          1h
kube-dns-v17.1-3pyz0                                               3/3       Running   0          1h
kube-dns-v17.1-zaskz                                               3/3       Running   0          1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt              1/1       Unknown   0          1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-mgst              1/1       Running   0          1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0              1/1       Running   0          1h
kubernetes-dashboard-3697774758-n808h                              1/1       Running   0          1h
kubernetes-dashboard-v1.1.1-ljtj5                                  1/1       Running   0          1h
l7-default-backend-2234341178-vo5z1                                1/1       Running   0          1h
l7-default-backend-v1.0-qe9yo                                      1/1       Running   0          1h"><pre class="notranslate">$ <span class="pl-s1">kubectl get rc --namespace=kube-system                                                   </span>
<span class="pl-c1">NAME                          DESIRED   CURRENT   READY     AGE</span>
<span class="pl-c1">kube-dns-v17.1                2         2         2         1h</span>
<span class="pl-c1">kubernetes-dashboard-v1.1.1   1         1         1         1h</span>
<span class="pl-c1">l7-default-backend-v1.0       1         1         1         1h</span>

$ <span class="pl-s1">kubectl get deployment --namespace=kube-system                                           </span>
<span class="pl-c1">NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE</span>
<span class="pl-c1">heapster-v1.1.0        1         1         1            1           1h</span>
<span class="pl-c1">heapster-v1.2.0        1         1         1            1           1h</span>
<span class="pl-c1">kube-dns               1         1         1            1           1h</span>
<span class="pl-c1">kubernetes-dashboard   1         1         1            1           1h</span>
<span class="pl-c1">l7-default-backend     1         1         1            1           1h</span>

$ <span class="pl-s1">kubectl get pods --namespace=kube-system                                                 </span>
<span class="pl-c1">NAME                                                               READY     STATUS    RESTARTS   AGE</span>
<span class="pl-c1">fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt   1/1       Unknown   0          1h</span>
<span class="pl-c1">fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-mgst   1/1       Running   0          1h</span>
<span class="pl-c1">fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0   1/1       Running   0          1h</span>
<span class="pl-c1">heapster-v1.1.0-2096339923-39key                                   2/2       Running   0          1h</span>
<span class="pl-c1">heapster-v1.2.0-2168613315-1bcy3                                   2/2       Running   0          1h</span>
<span class="pl-c1">kube-dns-4101612645-78hx6                                          4/4       Running   0          1h</span>
<span class="pl-c1">kube-dns-v17.1-3pyz0                                               3/3       Running   0          1h</span>
<span class="pl-c1">kube-dns-v17.1-zaskz                                               3/3       Running   0          1h</span>
<span class="pl-c1">kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt              1/1       Unknown   0          1h</span>
<span class="pl-c1">kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-mgst              1/1       Running   0          1h</span>
<span class="pl-c1">kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0              1/1       Running   0          1h</span>
<span class="pl-c1">kubernetes-dashboard-3697774758-n808h                              1/1       Running   0          1h</span>
<span class="pl-c1">kubernetes-dashboard-v1.1.1-ljtj5                                  1/1       Running   0          1h</span>
<span class="pl-c1">l7-default-backend-2234341178-vo5z1                                1/1       Running   0          1h</span>
<span class="pl-c1">l7-default-backend-v1.0-qe9yo                                      1/1       Running   0          1h</span></pre></div>
<p dir="auto">I found this as a counting error in <a href="https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster/337" rel="nofollow">https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster/337</a>. At first, I just thought it was incorrectly counting, and I attempted to fix that <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="189872287" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36924" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/36924/hovercard" href="https://github.com/kubernetes/kubernetes/pull/36924">#36924</a>. That fix is still valid and an improvement, but the underlying problem of multiple versions of addons still running is probably bad.</p>
<p dir="auto">We need a mechanism to delete the old ReplicationControllers/Deployments after an upgrade.</p>
<p dir="auto">There are a number of small pieces of discussion scattered across this and the rkt repository related to how to handle networking setup; I'd like to consolidate this discussion here, talk about all the issues and options I know about, and hopefully figure out what the best option is.</p>
<p dir="auto">Discussion refs:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="138331624" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2249" data-hovercard-type="issue" data-hovercard-url="/rkt/rkt/issues/2249/hovercard" href="https://github.com/rkt/rkt/issues/2249">rkt/rkt#2249</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="156300306" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26081" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/26081/hovercard" href="https://github.com/kubernetes/kubernetes/issues/26081">#26081</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="153996692" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/25404" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/25404/hovercard?comment_id=218402169&amp;comment_type=issue_comment" href="https://github.com/kubernetes/kubernetes/issues/25404#issuecomment-218402169">#25404 (comment)</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="150472538" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/24688" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/24688/hovercard" href="https://github.com/kubernetes/kubernetes/pull/24688">#24688</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="136210275" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/21923" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/21923/hovercard" href="https://github.com/kubernetes/kubernetes/issues/21923">#21923</a><br>
And quite possibly several more I missed</p>
<h2 dir="auto">Introduction</h2>
<p dir="auto">Kubernetes and rkt both make use of CNI to configure container network namespaces. Each has independent means of configuring and executing plugins. Kubernetes also has a number of features that relate somewhat to the network namespace, such as availability of the pod's IP in a number of locations (status, downwards API, etc).</p>
<p dir="auto">Until recently, rkt's handled network configuration on its own with no regard for Kubernetes' network logic. As of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="152677957" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/25062" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/25062/hovercard" href="https://github.com/kubernetes/kubernetes/pull/25062">#25062</a>, it was changed such that kubelet handles network setup and configuration for rkt. Unfortunately, both of these solutions have problems.</p>
<p dir="auto">Below, the problems with each approach are listed. Finally, some paths forwards are suggested.</p>
<h2 dir="auto">Issues</h2>
<ul dir="auto">
<li>The PodIP should be available to the kubelet prior to applications running for a number of features to work
<ul dir="auto">
<li>Downwards API (both volume and environment variables) rely on this</li>
<li><code class="notranslate">/etc/hosts</code> file, created by kubelet, relies on this (alternately, rkt should create this correctly)</li>
</ul>
</li>
<li>The network plugin configuration used by the docker container runtime should work, in general, for rkt with no changes (including filepath) (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="138331624" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2249" data-hovercard-type="issue" data-hovercard-url="/rkt/rkt/issues/2249/hovercard" href="https://github.com/rkt/rkt/issues/2249">rkt/rkt#2249</a>)</li>
<li>The solution should work with any rkt stage1 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="156300306" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26081" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/26081/hovercard" href="https://github.com/kubernetes/kubernetes/issues/26081">#26081</a>)</li>
</ul>
<h2 dir="auto">Solutions</h2>
<ol dir="auto">
<li>Rkt manages the network</li>
<li>Kubelet manages the network</li>
<li>Hybrid of 1 &amp; 2</li>
</ol>
<h3 dir="auto">Rkt manages networking</h3>
<p dir="auto">This option has a few problems:</p>
<ol dir="auto">
<li>The PodIP is not available to the kubelet until after the application is finalized (including mounts and environment variables) (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="136232605" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2223" data-hovercard-type="issue" data-hovercard-url="/rkt/rkt/issues/2223/hovercard" href="https://github.com/rkt/rkt/issues/2223">rkt/rkt#2223</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="145801751" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2375" data-hovercard-type="issue" data-hovercard-url="/rkt/rkt/issues/2375/hovercard" href="https://github.com/rkt/rkt/issues/2375">rkt/rkt#2375</a>)</li>
<li>The network plugin configuration path is not configurable (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="142424014" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2312" data-hovercard-type="pull_request" data-hovercard-url="/rkt/rkt/pull/2312/hovercard" href="https://github.com/rkt/rkt/pull/2312">rkt/rkt#2312</a>)</li>
<li>Duplication of logic/code between kubelet and rkt (more or less depending on who creates <code class="notranslate">/etc/hosts</code> and so on).</li>
</ol>
<p dir="auto">The first issue there is a difficult one to solve and is the main reason that this approach was discarded, whether temporarily or not.</p>
<p dir="auto">This is probably the only robust long-term solution when considering the large variety of stage1's (and thus network configurations) that are available and, once rkt provides a container-level interface (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="145801751" data-permission-text="Title is private" data-url="https://github.com/rkt/rkt/issues/2375" data-hovercard-type="issue" data-hovercard-url="/rkt/rkt/issues/2375/hovercard" href="https://github.com/rkt/rkt/issues/2375">rkt/rkt#2375</a>), there should be a natural way to solve problem number one via the separation of sandbox creation and container creation.</p>
<h3 dir="auto">Kubelet manages networking</h3>
<p dir="auto">This option has its own problems:</p>
<ol dir="auto">
<li>Does not work with kvm flavor stage1 (or any stage1 that does not use Linux network namespaces or does not support <code class="notranslate">--net=host</code>).</li>
<li>Depends on availability of nsenter / ip tools (already depended on elsewhere I believe, mostly a nonissue)</li>
<li>Results in the <code class="notranslate">rkt</code> cli reporting incorrect network information</li>
</ol>
<p dir="auto">This allows fulfilling more of kubelet's assumptions about networking, but does not play nicely with rkt's existing abstractions. It does solve the most problems for the case of the default stage1 and existing rkt featureset while also not being a large amount of code. It also shares significant logic with the docker network setup, with the benefits that entails.</p>
<h3 dir="auto">Hybrid</h3>
<p dir="auto">The kubelet could decide whether it or rkt should manage the networking based on whether the stage1 suports it. When rkt manages networking, the features dependent on PodIP being available early on will not work.</p>
<ol dir="auto">
<li>More code in kubelet to support both methods</li>
<li>rkt does not provide an easy means to determine whether one or the other is appropriate currently</li>
<li>Inconsistent featureset depending on stage1 (<g-emoji class="g-emoji" alias="frowning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f626.png">😦</g-emoji>)</li>
</ol>
<h2 dir="auto">Conclusion</h2>
<p dir="auto">My current preference is to continue with the kubelet-managed network for 1.0 (though given the timeline, I'm not sure there's another choice), and rethink this afterwards. If it's possible to support a hybrid approach by 1.0 with the understanding that it's a temporary solution, that also seems reasonable.</p>
<p dir="auto">Ideally, the kubelet's abstractions would not interfere with rkt's ability to manage networking itself since, I admit, the above solution is a bit of a hacky one.</p>
<p dir="auto">It's also possible that the upcoming Container Runtime Interface changes, with a focus on sandboxes, will require further changes in both the kubelet and rkt that changes what issues are present.</p>
<p dir="auto">I'd like to gather other's thoughts and make sure we're all on the same page and figure out if there are other options or issues I'm not thinking of.</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yifan-gu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yifan-gu">@yifan-gu</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jellonek/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jellonek">@jellonek</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pskrzyns/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pskrzyns">@pskrzyns</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jonboulle/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jonboulle">@jonboulle</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dcbw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dcbw">@dcbw</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/iaguis/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/iaguis">@iaguis</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/philips/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/philips">@philips</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/freehan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/freehan">@freehan</a></p>