<p dir="auto">I got an issue with controller-manager starting from v1.2.0-beta.1 and it still persists in v1.2.0 release.</p>
<p dir="auto">It keeps restarting pods kube-apiserver, kube-controller-manager, kube-proxy and kube-scheduler every 30s :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME                                                     READY     STATUS    RESTARTS   AGE
...
kube-apiserver-ip-10-242-128-234.ec2.internal            1/1       Running   0          12s
kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       Running   0          16s
kube-proxy-ip-10-242-128-234.ec2.internal                1/1       Running   0          16s
kube-scheduler-ip-10-242-128-234.ec2.internal            1/1       Running   0          15s"><pre class="notranslate"><code class="notranslate">NAME                                                     READY     STATUS    RESTARTS   AGE
...
kube-apiserver-ip-10-242-128-234.ec2.internal            1/1       Running   0          12s
kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       Running   0          16s
kube-proxy-ip-10-242-128-234.ec2.internal                1/1       Running   0          16s
kube-scheduler-ip-10-242-128-234.ec2.internal            1/1       Running   0          15s
</code></pre></div>
<p dir="auto">Age for the listed pods don't go above 30s.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME                                                     READY     STATUS    RESTARTS   AGE
....
kube-apiserver-ip-10-242-128-234.ec2.internal            0/1       Pending   0          1s
kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       Running   0          1s
kube-scheduler-ip-10-242-128-234.ec2.internal            0/1       Pending   0          1s"><pre class="notranslate"><code class="notranslate">NAME                                                     READY     STATUS    RESTARTS   AGE
....
kube-apiserver-ip-10-242-128-234.ec2.internal            0/1       Pending   0          1s
kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       Running   0          1s
kube-scheduler-ip-10-242-128-234.ec2.internal            0/1       Pending   0          1s
</code></pre></div>
<p dir="auto">If i switch controller-manager to v1.2.0-beta.0 issue dissapear.</p>
<p dir="auto">The strange thing is that docker containers are keep running</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
....
c44ff71caf21        gcr.io/google_containers/hyperkube:v1.2.0   &quot;/hyperkube scheduler&quot;   33 minutes ago      Up 33 minutes                           k8s_kube-scheduler.7b10907e_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_ed62caa9
d8ef6ef809ca        gcr.io/google_containers/hyperkube:v1.2.0   &quot;/hyperkube proxy --m&quot;   33 minutes ago      Up 33 minutes                           k8s_kube-proxy.c9b75a4a_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_3ee808aa
068a6ab6793a        gcr.io/google_containers/hyperkube:v1.2.0   &quot;/hyperkube controlle&quot;   33 minutes ago      Up 33 minutes                           k8s_kube-controller-manager.2e450a4c_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_8db26042
76e4cda29efb        gcr.io/google_containers/hyperkube:v1.2.0   &quot;/hyperkube apiserver&quot;   33 minutes ago      Up 33 minutes                           k8s_kube-apiserver.f4b25f9b_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_471bfae7
11eb022a4939        gcr.io/google_containers/pause:2.0          &quot;/pause&quot;                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_a51591f2
a1d9d564218c        gcr.io/google_containers/pause:2.0          &quot;/pause&quot;                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_6accda61
1a1a278f12d5        gcr.io/google_containers/pause:2.0          &quot;/pause&quot;                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_209f305d
48e6e4e4d4bc        gcr.io/google_containers/pause:2.0          &quot;/pause&quot;                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_3b289a47"><pre class="notranslate"><code class="notranslate">CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
....
c44ff71caf21        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube scheduler"   33 minutes ago      Up 33 minutes                           k8s_kube-scheduler.7b10907e_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_ed62caa9
d8ef6ef809ca        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube proxy --m"   33 minutes ago      Up 33 minutes                           k8s_kube-proxy.c9b75a4a_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_3ee808aa
068a6ab6793a        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube controlle"   33 minutes ago      Up 33 minutes                           k8s_kube-controller-manager.2e450a4c_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_8db26042
76e4cda29efb        gcr.io/google_containers/hyperkube:v1.2.0   "/hyperkube apiserver"   33 minutes ago      Up 33 minutes                           k8s_kube-apiserver.f4b25f9b_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_471bfae7
11eb022a4939        gcr.io/google_containers/pause:2.0          "/pause"                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_a51591f2
a1d9d564218c        gcr.io/google_containers/pause:2.0          "/pause"                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_6accda61
1a1a278f12d5        gcr.io/google_containers/pause:2.0          "/pause"                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_209f305d
48e6e4e4d4bc        gcr.io/google_containers/pause:2.0          "/pause"                 33 minutes ago      Up 33 minutes                           k8s_POD.6059dfa2_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_3b289a47
</code></pre></div>
<p dir="auto">Error log from controller-manager:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0322 15:57:36.787253       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.787288       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:57:36.817601       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.817644       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:57:36.819261       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found
E0322 15:57:36.828904       1 nodecontroller.go:352] pods &quot;kube-scheduler-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:57:37.659569       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:37.659586       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:57:38.645497       1 nodecontroller.go:352] pods &quot;kube-scheduler-ip-10-242-128-234.ec2.internal&quot; not found
E0322 15:57:38.678738       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:57:39.601407       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:39.601439       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:57:40.619997       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:40.620013       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:06.283648       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.295400       1 nodecontroller.go:352] pods &quot;kube-scheduler-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:06.372126       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.404804       1 nodecontroller.go:352] pods &quot;kube-proxy-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:06.442257       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.458821       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:06.527879       1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.528300       1 nodecontroller.go:352] pods &quot;kube-apiserver-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:06.694718       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.694758       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:06.777777       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.777819       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:58:06.788130       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:36.558947       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.573911       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:36.614092       1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.615441       1 nodecontroller.go:352] pods &quot;kube-apiserver-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:36.653430       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
I0322 15:58:36.665033       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.665048       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:36.736231       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.744630       1 nodecontroller.go:352] pods &quot;kube-proxy-ip-10-242-128-234.ec2.internal&quot; not found
I0322 15:58:36.786456       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.786470       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:58:37.743522       1 nodecontroller.go:352] pods &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal&quot; not found"><pre class="notranslate"><code class="notranslate">I0322 15:57:36.787253       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.787288       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:57:36.817601       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.817644       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:57:36.819261       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
E0322 15:57:36.828904       1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
I0322 15:57:37.659569       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:37.659586       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:57:38.645497       1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
E0322 15:57:38.678738       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:57:39.601407       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:39.601439       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:57:40.619997       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:40.620013       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:06.283648       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.295400       1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.372126       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.404804       1 nodecontroller.go:352] pods "kube-proxy-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.442257       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.458821       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.527879       1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.528300       1 nodecontroller.go:352] pods "kube-apiserver-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.694718       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.694758       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:06.777777       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.777819       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:58:06.788130       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.558947       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.573911       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.614092       1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.615441       1 nodecontroller.go:352] pods "kube-apiserver-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.653430       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
I0322 15:58:36.665033       1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.665048       1 nodecontroller.go:352] &lt;nil&gt;
I0322 15:58:36.736231       1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.744630       1 nodecontroller.go:352] pods "kube-proxy-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.786456       1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.786470       1 nodecontroller.go:352] &lt;nil&gt;
E0322 15:58:37.743522       1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
</code></pre></div>
<p dir="auto">Error log from kubelet:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Mar 22 16:01:38 master kubelet[1493]: E0322 16:01:38.917605    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.195729    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(4a5d3ea1-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196138    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(4a39ac2f-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196300    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197232    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5c68bb91-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197419    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(5c78f49f-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197556    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(5c4a0855-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:40 master kubelet[1493]: W0322 16:01:40.395965    1493 request.go:627] Throttling request took 110.236442ms, request: POST:http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods
Mar 22 16:01:41 master kubelet[1493]: E0322 16:01:41.199445    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5ca9cde4-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:42 master kubelet[1493]: E0322 16:01:42.201111    1493 kubelet.go:1753] Deleting mirror pod &quot;kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5d393e7a-f047-11e5-a5d4-0af71f25e715)&quot; because it is outdated
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.619715    1493 kubelet.go:2741] Error updating node status, will retry: error getting node &quot;ip-10-242-128-234.ec2.internal&quot;: nodes &quot;ip-10-242-128-234.ec2.internal&quot; not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.655629    1493 kubelet.go:2741] Error updating node status, will retry: error getting node &quot;ip-10-242-128-234.ec2.internal&quot;: nodes &quot;ip-10-242-128-234.ec2.internal&quot; not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.656954    1493 kubelet.go:2741] Error updating node status, will retry: error getting node &quot;ip-10-242-128-234.ec2.internal&quot;: nodes &quot;ip-10-242-128-234.ec2.internal&quot; not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.658257    1493 kubelet.go:2741] Error updating node status, will retry: error getting node &quot;ip-10-242-128-234.ec2.internal&quot;: nodes &quot;ip-10-242-128-234.ec2.internal&quot; not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659539    1493 kubelet.go:2741] Error updating node status, will retry: error getting node &quot;ip-10-242-128-234.ec2.internal&quot;: nodes &quot;ip-10-242-128-234.ec2.internal&quot; not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659553    1493 kubelet.go:1183] Unable to update node status: update node status exceeds retry count"><pre class="notranslate"><code class="notranslate">Mar 22 16:01:38 master kubelet[1493]: E0322 16:01:38.917605    1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.195729    1493 kubelet.go:1753] Deleting mirror pod "kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(4a5d3ea1-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196138    1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(4a39ac2f-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196300    1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197232    1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5c68bb91-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197419    1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(5c78f49f-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197556    1493 kubelet.go:1753] Deleting mirror pod "kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(5c4a0855-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: W0322 16:01:40.395965    1493 request.go:627] Throttling request took 110.236442ms, request: POST:http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods
Mar 22 16:01:41 master kubelet[1493]: E0322 16:01:41.199445    1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5ca9cde4-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:42 master kubelet[1493]: E0322 16:01:42.201111    1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5d393e7a-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.619715    1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.655629    1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.656954    1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.658257    1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659539    1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659553    1493 kubelet.go:1183] Unable to update node status: update node status exceeds retry count
</code></pre></div>
<p dir="auto">My understanding is that kubelet thinks that these pods are duplicates and tries to remove them, but at the same time doesn't touch real docker containers.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): vsphere, ResourcePool, ComputeResource</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug Report</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5+coreos.0", GitCommit:"f70c2e5b2944cb5d622621a706bdec3d8a5a9c5e", GitTreeState:"clean", BuildDate:"2016-10-31T19:16:47Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>: CoreOS on vSphere</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: vSphere</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1185.3.0 (MoreOS)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux k8-m1 4.7.3-coreos-r2 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Tue Nov 1 01:38:43 UTC 2016</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I configured the controller-manager to use vsphere as the cloud provider and supplied it with a provider cloud configuration. When the controller-manager pod started, it panicked with the following stack trace;</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-11-06T00:34:22.670422172Z panic: reflect.Set: value of type mo.ResourcePool is not assignable to type mo.ComputeResource
2016-11-06T00:34:22.670477454Z 
2016-11-06T00:34:22.670494551Z goroutine 68 [running]:
2016-11-06T00:34:22.670592707Z panic(0x38d5200, 0xc820d2b310)
2016-11-06T00:34:22.670754360Z 	/usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-11-06T00:34:22.670990355Z reflect.Value.assignTo(0x4a676e0, 0xc8200a5600, 0x99, 0x4df7650, 0xb, 0x4a66e60, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.671019041Z 	/usr/local/go/src/reflect/value.go:2164 +0x3be
2016-11-06T00:34:22.671380809Z reflect.Value.Set(0x4a66e60, 0xc82049e780, 0x199, 0x4a676e0, 0xc8200a5600, 0x99)
2016-11-06T00:34:22.671462864Z 	/usr/local/go/src/reflect/value.go:1334 +0x95
2016-11-06T00:34:22.671715875Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo.LoadRetrievePropertiesResponse(0xc820d308e0, 0x46ecda0, 0xc82049e780, 0x0, 0x0)
2016-11-06T00:34:22.672055230Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo/retrieve.go:128 +0xe21
2016-11-06T00:34:22.672431372Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).Retrieve(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820d3ca80, 0x1, 0x1, 0xc820cecfd0, 0x1, 0x1, 0x46ecda0, ...)
2016-11-06T00:34:22.672467148Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:167 +0x52f
2016-11-06T00:34:22.672606970Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).RetrieveOne(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, 0xc, 0xc820cec9e0, 0xa, 0xc820cecfd0, 0x1, 0x1, ...)
2016-11-06T00:34:22.672681600Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:173 +0x10e
2016-11-06T00:34:22.672871709Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object.Common.Properties(0x0, 0x0, 0xc820b1e500, 0xc8202cedd0, 0xb, 0xc8202cee00, 0xb, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, ...)
2016-11-06T00:34:22.672904730Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/common.go:97 +0x19f
2016-11-06T00:34:22.673072861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.readInstance(0xc82025b0e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673104582Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:223 +0xe8f
2016-11-06T00:34:22.673118861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.newVSphere(0xc820252200, 0x18, 0xc8204c25f0, 0xc, 0xc82011fd60, 0x9, 0xc8204c2048, 0x3, 0x1, 0xc8204c2ae8, ...)
2016-11-06T00:34:22.673128111Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:237 +0x7e
2016-11-06T00:34:22.673204822Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.init.1.func1(0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673274571Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:153 +0xdf
2016-11-06T00:34:22.673365023Z k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffc9a344abf, 0x7, 0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673457368Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:62 +0x112
2016-11-06T00:34:22.673526314Z k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffc9a344abf, 0x7, 0x7ffc9a344ad6, 0x1c, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673582280Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:84 +0x3e2
2016-11-06T00:34:22.673593710Z k8s.io/kubernetes/cmd/kube-controller-manager/app.StartControllers(0xc82062f900, 0xc8203aea80, 0xc8201de340, 0xc8203af860, 0x7f8a572d6790, 0xc820418640, 0x0, 0x0)
2016-11-06T00:34:22.673674949Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:225 +0x719
2016-11-06T00:34:22.673686774Z k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2(0xc8203af860)
2016-11-06T00:34:22.673696396Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:166 +0x6a
2016-11-06T00:34:22.673755265Z created by k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run
2016-11-06T00:34:22.673766424Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91"><pre class="notranslate"><code class="notranslate">2016-11-06T00:34:22.670422172Z panic: reflect.Set: value of type mo.ResourcePool is not assignable to type mo.ComputeResource
2016-11-06T00:34:22.670477454Z 
2016-11-06T00:34:22.670494551Z goroutine 68 [running]:
2016-11-06T00:34:22.670592707Z panic(0x38d5200, 0xc820d2b310)
2016-11-06T00:34:22.670754360Z 	/usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-11-06T00:34:22.670990355Z reflect.Value.assignTo(0x4a676e0, 0xc8200a5600, 0x99, 0x4df7650, 0xb, 0x4a66e60, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.671019041Z 	/usr/local/go/src/reflect/value.go:2164 +0x3be
2016-11-06T00:34:22.671380809Z reflect.Value.Set(0x4a66e60, 0xc82049e780, 0x199, 0x4a676e0, 0xc8200a5600, 0x99)
2016-11-06T00:34:22.671462864Z 	/usr/local/go/src/reflect/value.go:1334 +0x95
2016-11-06T00:34:22.671715875Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo.LoadRetrievePropertiesResponse(0xc820d308e0, 0x46ecda0, 0xc82049e780, 0x0, 0x0)
2016-11-06T00:34:22.672055230Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo/retrieve.go:128 +0xe21
2016-11-06T00:34:22.672431372Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).Retrieve(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820d3ca80, 0x1, 0x1, 0xc820cecfd0, 0x1, 0x1, 0x46ecda0, ...)
2016-11-06T00:34:22.672467148Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:167 +0x52f
2016-11-06T00:34:22.672606970Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).RetrieveOne(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, 0xc, 0xc820cec9e0, 0xa, 0xc820cecfd0, 0x1, 0x1, ...)
2016-11-06T00:34:22.672681600Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:173 +0x10e
2016-11-06T00:34:22.672871709Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object.Common.Properties(0x0, 0x0, 0xc820b1e500, 0xc8202cedd0, 0xb, 0xc8202cee00, 0xb, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, ...)
2016-11-06T00:34:22.672904730Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/common.go:97 +0x19f
2016-11-06T00:34:22.673072861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.readInstance(0xc82025b0e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673104582Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:223 +0xe8f
2016-11-06T00:34:22.673118861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.newVSphere(0xc820252200, 0x18, 0xc8204c25f0, 0xc, 0xc82011fd60, 0x9, 0xc8204c2048, 0x3, 0x1, 0xc8204c2ae8, ...)
2016-11-06T00:34:22.673128111Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:237 +0x7e
2016-11-06T00:34:22.673204822Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.init.1.func1(0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673274571Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:153 +0xdf
2016-11-06T00:34:22.673365023Z k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffc9a344abf, 0x7, 0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673457368Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:62 +0x112
2016-11-06T00:34:22.673526314Z k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffc9a344abf, 0x7, 0x7ffc9a344ad6, 0x1c, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673582280Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:84 +0x3e2
2016-11-06T00:34:22.673593710Z k8s.io/kubernetes/cmd/kube-controller-manager/app.StartControllers(0xc82062f900, 0xc8203aea80, 0xc8201de340, 0xc8203af860, 0x7f8a572d6790, 0xc820418640, 0x0, 0x0)
2016-11-06T00:34:22.673674949Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:225 +0x719
2016-11-06T00:34:22.673686774Z k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2(0xc8203af860)
2016-11-06T00:34:22.673696396Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:166 +0x6a
2016-11-06T00:34:22.673755265Z created by k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run
2016-11-06T00:34:22.673766424Z 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
I expected the controller-manager pod to run as normal.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Configure the controller-manager pod to use the vsphere cloud provider and pass the following provider cloud configuration file;</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[Global]
server = vcenter
port = 443
user = administrator@vsphere.local
password = removed
insecure-flag = true
datacenter = FC

[Network]
public-network = External"><pre class="notranslate"><code class="notranslate">[Global]
server = vcenter
port = 443
user = administrator@vsphere.local
password = removed
insecure-flag = true
datacenter = FC

[Network]
public-network = External
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<ul dir="auto">
<li>This cluster was created by following the <a href="https://coreos.com/kubernetes/docs/latest/getting-started.html" rel="nofollow">CoreOS + Kubernetes Step by Step</a> guide. The only deviation from the guide was to adjust the /etc/kubernetes/manifests/kube-controller-manager.yaml file to include configuration flags for cloud-provider and cloud-config, and adding an additional volume and volume mount for the provider cloud config file.</li>
<li>The CoreOS VMs are part of a single resource group (K8) on the same ESXi host.</li>
<li>My base image for all of the hyperkube containers is quay.io/coreos/hyperkube:v1.4.5_coreos.0</li>
<li>Before adding the additional configuration to use the vsphere cloud provider, this cluster was working as expected. The only change that was made to produce the stack trace above was to enable the vsphere cloud provider and pass the provider cloud config above.</li>
<li>I am still fairly new to Kubernetes so i'm open to the possibility that I might have made a fatal error somewhere, but as far as I can tell this does appear to be a genuine bug. Apologies if this does turn out to be a mistake on my behalf!</li>
</ul>