<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">kubeadm join<br>
kubeadm join crash<br>
kubeadm close of closed channel</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>kubeadm version</strong> (use <code class="notranslate">kubeadm version</code>):</p>
<p dir="auto">kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+85fe0f1aadf91e", GitCommit:"85fe0f1aadf91e134102cf3c01a9eed11a7e257f", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CentOS</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 3.10.0-327.36.1.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: yum</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">kubeadm crash when join mulitple API endpoints</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">join mulitple API endpoints</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">kubeadm output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# kubeadm join --token=ab6000.d5be12581d4bc52d 10.37.0.8
Running pre-flight checks
&lt;util/tokens&gt; validating provided token
&lt;node/discovery&gt; created cluster info discovery client, requesting info from &quot;http://10.37.0.8:9898/cluster-info/v1/?token-id=ab6000&quot;
&lt;node/discovery&gt; cluster info object received, verifying signature using given token
&lt;node/discovery&gt; cluster info signature and contents are valid, will use API endpoints [https://10.37.0.8:6443 https://10.37.0.10:6443 https://10.37.0.11:6443 https://10.37.0.12:6443]
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.8:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.10:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.11:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.12:6443
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; successfully established connection with endpoint https://10.37.0.11:6443
&lt;node/csr&gt; created API client to obtain unique certificate for this node, generating keys and certificate signing request
&lt;node/bootstrap&gt; successfully established connection with endpoint https://10.37.0.12:6443
E1117 19:59:42.617610    3072 runtime.go:64] Observed a panic: &quot;close of closed channel&quot; (close of closed channel)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
/usr/local/go/src/runtime/asm_amd64.s:479
/usr/local/go/src/runtime/panic.go:458
/usr/local/go/src/runtime/chan.go:311
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:83
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:88
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:90
/usr/local/go/src/runtime/asm_amd64.s:2086
panic: close of closed channel [recovered]
	panic: close of closed channel

goroutine 14 [running]:
panic(0x2065ce0, 0xc42033b990)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126
panic(0x2065ce0, 0xc42033b990)
	/usr/local/go/src/runtime/panic.go:458 +0x243
k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection.func1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:83 +0x29d
k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc4205f3ed0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0x5e
k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc4205f3ed0, 0x12a05f200, 0x0, 0xc420272601, 0xc4203e8fc0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:88 +0xad
k8s.io/kubernetes/pkg/util/wait.Until(0xc4205f3ed0, 0x12a05f200, 0xc4203e8fc0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x4d
k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection.func1(0xc42044a6b0, 0xc4201ea180, 0xc4203e8fc0, 0xc4203e9020, 0xc420172000, 0x401, 0x480, 0xc42044a6a0, 0x4, 0xc4203edca0, ...)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:90 +0x100
created by k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:91 +0x3ed"><pre class="notranslate"><code class="notranslate"># kubeadm join --token=ab6000.d5be12581d4bc52d 10.37.0.8
Running pre-flight checks
&lt;util/tokens&gt; validating provided token
&lt;node/discovery&gt; created cluster info discovery client, requesting info from "http://10.37.0.8:9898/cluster-info/v1/?token-id=ab6000"
&lt;node/discovery&gt; cluster info object received, verifying signature using given token
&lt;node/discovery&gt; cluster info signature and contents are valid, will use API endpoints [https://10.37.0.8:6443 https://10.37.0.10:6443 https://10.37.0.11:6443 https://10.37.0.12:6443]
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.8:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.10:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.11:6443
&lt;node/bootstrap&gt; trying to connect to endpoint https://10.37.0.12:6443
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; detected server version v1.4.5
&lt;node/bootstrap&gt; successfully established connection with endpoint https://10.37.0.11:6443
&lt;node/csr&gt; created API client to obtain unique certificate for this node, generating keys and certificate signing request
&lt;node/bootstrap&gt; successfully established connection with endpoint https://10.37.0.12:6443
E1117 19:59:42.617610    3072 runtime.go:64] Observed a panic: "close of closed channel" (close of closed channel)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
/usr/local/go/src/runtime/asm_amd64.s:479
/usr/local/go/src/runtime/panic.go:458
/usr/local/go/src/runtime/chan.go:311
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:83
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:88
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:90
/usr/local/go/src/runtime/asm_amd64.s:2086
panic: close of closed channel [recovered]
	panic: close of closed channel

goroutine 14 [running]:
panic(0x2065ce0, 0xc42033b990)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126
panic(0x2065ce0, 0xc42033b990)
	/usr/local/go/src/runtime/panic.go:458 +0x243
k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection.func1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:83 +0x29d
k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc4205f3ed0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0x5e
k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc4205f3ed0, 0x12a05f200, 0x0, 0xc420272601, 0xc4203e8fc0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:88 +0xad
k8s.io/kubernetes/pkg/util/wait.Until(0xc4205f3ed0, 0x12a05f200, 0xc4203e8fc0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x4d
k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection.func1(0xc42044a6b0, 0xc4201ea180, 0xc4203e8fc0, 0xc4203e9020, 0xc420172000, 0x401, 0x480, 0xc42044a6a0, 0x4, 0xc4203edca0, ...)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:90 +0x100
created by k8s.io/kubernetes/cmd/kubeadm/app/node.EstablishMasterConnection
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/node/bootstrap.go:91 +0x3ed
</code></pre></div>
<p dir="auto">We want to deprecate today's debian-based container-vm image, and switch to GCI image by default. Before we do that, we need to validate the image using Kubernetes testing infrastructure. GCI image has three different release channels: dev, beta, and stable. Besides that they also build ci release nightly. Today we already run the the following jenkins jobs against each image release:</p>
<ul dir="auto">
<li>kubernetes-gce-e2e,</li>
<li>kubernetes-gce-e2e-slow,</li>
<li>kubernetes-gce-e2e-serial</li>
</ul>
<p dir="auto">But we are missing kubernetes-gce-e2e-soak, kubernetes-gce-e2e-scalability, probably also kubernetes-gce-e2e-upgrade (? not sure how to handle this since it is the first release), and kubernetes-gce-e2e-features.</p>
<p dir="auto">We also need to include gci image to node-e2e test suite too.</p>
<p dir="auto">Next step we should to identify the critical builds for gci to block merge-bot.</p>
<p dir="auto">Eventually, we can switch GCI image as the default image on GCE, but keep debian-based container-vm image testing until we can deprecate it.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/andyzheng0831/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/andyzheng0831">@andyzheng0831</a> and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wonderfly/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wonderfly">@wonderfly</a> are going to write detail plan on this.</p>
<p dir="auto">cc/ <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/roberthbailey/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/roberthbailey">@roberthbailey</a> from GKE side.<br>
cc/ <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mansoorj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mansoorj">@mansoorj</a> from PM side</p>