<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">concurrent map writes</p>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">$ kubectl version<br>
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Openstack</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
$ cat /etc/os-release<br>
NAME="Ubuntu"<br>
VERSION="16.04.1 LTS (Xenial Xerus)"<br>
ID=ubuntu<br>
ID_LIKE=debian<br>
PRETTY_NAME="Ubuntu 16.04.1 LTS"<br>
VERSION_ID="16.04"<br>
HOME_URL="<a href="http://www.ubuntu.com/" rel="nofollow">http://www.ubuntu.com/</a>"<br>
SUPPORT_URL="<a href="http://help.ubuntu.com/" rel="nofollow">http://help.ubuntu.com/</a>"<br>
BUG_REPORT_URL="<a href="http://bugs.launchpad.net/ubuntu/" rel="nofollow">http://bugs.launchpad.net/ubuntu/</a>"<br>
UBUNTU_CODENAME=xenial</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
$ uname -a<br>
Linux Master1 4.4.0-34-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35443479" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/53" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/53/hovercard" href="https://github.com/kubernetes/kubernetes/pull/53">#53</a>-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I noticed a panic in the kube-controller-manager in a 3 node HA Kubernetes controller deployment. Here are the logs:<br>
Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: E0830 15:36:57.457558   19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the<br>
Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.741242   19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this<br>
Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.762328   19004 reflector.go:334] pkg/controller/serviceaccount/tokens_controller.go:147: watch of _api.ServiceAccount<br>
Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.764460   19004 reflector.go:334] pkg/controller/serviceaccount/serviceaccounts_controller.go:142: watch of *api.Servi<br>
Aug 30 15:36:58 ubuntu16 kube-controller-manager[19004]: W0830 15:36:58.764857   19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this<br>
Aug 30 15:36:58 ubuntu16 kube-controller-manager[19004]: W0830 15:36:58.771127   19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this<br>
Aug 30 15:37:01 ubuntu16 kube-controller-manager[19004]: E0830 15:37:01.594976   19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the<br>
Aug 30 15:37:03 ubuntu16 kube-controller-manager[19004]: E0830 15:37:03.554584   19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: fatal error: concurrent map writes<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: goroutine 823 [running]:<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.throw(0x2c5d4f0, 0x15)<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]:         /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8208fcab0 sp=0xc8208fca98<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.mapassign1(0x1e3e820, 0xc8205518f0, 0xc8208fcbf8, 0xc8208fcc48)<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]:         /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8208fcb58 sp=0xc8208fcab0<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_DeltaFIFO).queueActionLocked(0xc8200dadc0, 0x2a3fcd0, 0x4, 0x29f02e0, 0xc82106c2c0, 0x0, 0x0)<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc8208fc<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_DeltaFIFO).Resync(0xc8200dadc0, 0x0, 0x0)<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc8208fc<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_Reflector).ListAndWatch.func1(0xc820107318, 0xc82004ede0, 0xc8202e23c0, 0xc821233aa0, 0xc820107<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc8208fcf<br>
Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.goexit()</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
I did not expect to hit a code path where I noticed concurrent map writes.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Used the following steps:<br>
<a href="https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-kubernetes-controller.md">https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-kubernetes-controller.md</a><br>
The difference is that I used Openstack as the cloud provider</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
I had cloned the 3 HA controller VMs from a single volume, so initially they ended up with the same Linux hostname. That explains the failure in the leader election, but not sure if it's related to the panic. I fixed the hostnames later and the leader election succeeded.</p>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
Feature request.</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
master</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
In Mesos 1.x, <a href="http://mesos.apache.org/documentation/latest/mesos-containerizer/" rel="nofollow">Mesos container</a> is the default container runtime of tasks; when running Kubernetes on Mesos (<a href="https://github.com/kubernetes-incubator/kube-mesos-framework/">kube-mesos</a>), we'd like to also using only one container runtime environment to reduce OP's maintenance effort.</p>
<p dir="auto">That's one project of <a href="https://github.com/kubernetes-incubator/kube-mesos-framework/">kube-mesos</a> ; but I do not want to duplicate Kubelet in <a href="https://github.com/kubernetes-incubator/kube-mesos-framework/">kube-mesos</a> project.</p>