<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li>AWS</li>
<li>Xenial</li>
<li>Terraform</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I am setting up a K8s cluster using Terraform + kubeadm in AWS.  Etcd, kube master, and all kube nodes are in auto-scaling groups and behind load balancers, with the idea being that if some or all nodes fail, the cluster will heal itself.  Most recently, I have been working to get the aws cloud-provider to work.  At one point, I corrected my kubelet systemd file and deleted the kmaster and knode instances.  When they came back up, the discovery service failed to start on the master node.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;master/discovery&gt; failed to create &quot;kube-discovery&quot; deployment [deployments.extensions &quot;kube-discovery&quot; already exists]"><pre class="notranslate"><code class="notranslate">&lt;master/discovery&gt; failed to create "kube-discovery" deployment [deployments.extensions "kube-discovery" already exists]
</code></pre></div>
<p dir="auto">With the existing deployment, I was surprised that a new discovery pod wasn't scheduled, but it stayed in pending.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="root@ip-10-253-128-238:~# kubectl describe pod/kube-discovery-1150918428-aca0v --namespace=kube-system
Name:       kube-discovery-1150918428-aca0v
Namespace:  kube-system
Node:       /
Labels:     component=kube-discovery
        k8s-app=kube-discovery
        kubernetes.io/cluster-service=true
        name=kube-discovery
        pod-template-hash=1150918428
        tier=node
Status:     Pending
IP:     
Controllers:    ReplicaSet/kube-discovery-1150918428
Containers:
  kube-discovery:
    Image:  gcr.io/google_containers/kube-discovery-amd64:1.0
    Port:   9898/TCP
    Command:
      /usr/local/bin/kube-discovery
    Volume Mounts:
      /tmp/secret from clusterinfo (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lhygh (ro)
    Environment Variables:  &lt;none&gt;
Conditions:
  Type      Status
  PodScheduled  False 
Volumes:
  clusterinfo:
    Type:   Secret (a volume populated by a Secret)
    SecretName: clusterinfo
  default-token-lhygh:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-lhygh
QoS Class:  BestEffort
Tolerations:    dedicated=master:NoSchedule
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  59m       20s     147 {default-scheduler }            Warning     FailedScheduling    pod (kube-discovery-1150918428-aca0v) failed to fit in any node
fit failure on node (ip-10-253-128-238.ec2.internal): MatchNodeSelector"><pre class="notranslate"><code class="notranslate">root@ip-10-253-128-238:~# kubectl describe pod/kube-discovery-1150918428-aca0v --namespace=kube-system
Name:       kube-discovery-1150918428-aca0v
Namespace:  kube-system
Node:       /
Labels:     component=kube-discovery
        k8s-app=kube-discovery
        kubernetes.io/cluster-service=true
        name=kube-discovery
        pod-template-hash=1150918428
        tier=node
Status:     Pending
IP:     
Controllers:    ReplicaSet/kube-discovery-1150918428
Containers:
  kube-discovery:
    Image:  gcr.io/google_containers/kube-discovery-amd64:1.0
    Port:   9898/TCP
    Command:
      /usr/local/bin/kube-discovery
    Volume Mounts:
      /tmp/secret from clusterinfo (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lhygh (ro)
    Environment Variables:  &lt;none&gt;
Conditions:
  Type      Status
  PodScheduled  False 
Volumes:
  clusterinfo:
    Type:   Secret (a volume populated by a Secret)
    SecretName: clusterinfo
  default-token-lhygh:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-lhygh
QoS Class:  BestEffort
Tolerations:    dedicated=master:NoSchedule
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  59m       20s     147 {default-scheduler }            Warning     FailedScheduling    pod (kube-discovery-1150918428-aca0v) failed to fit in any node
fit failure on node (ip-10-253-128-238.ec2.internal): MatchNodeSelector
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Either for kubeadm to delete the existing deployment, or for that deployment to sucessfully schedule new pods on the new master node.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Bring up a cluster using kubeadm and external etcd.  After the cluster is up, delete the master node and try to bring it back up (using the same command as the first time).</p>
<p dir="auto">From <a href="https://00e9e64bac6b200a32447dc4f63b6030019cc0bc3a44f16a27-apidata.googleusercontent.com/download/storage/v1_internal/b/kubernetes-jenkins/o/pr-logs%2Fpull%2F21439%2Fkubernetes-pull-test-unit-integration%2F14897%2Fbuild-log.txt?qk=AD5uMEvhvtd0nU5LWkcTBK9YLg-hvrEQTWAt9cK36vzC3Zy2HiqUXBVZSnxKHiCp67X0z0Xd-JeEOgnBtccHilLhlzARzAhmuSSrBVr6Sssz4EvF10NRGnV6OpUdMT4VHvSiR4wt_U8vT66ZBARu7PucUuFuQQVnROKuqqXqcmLLyfR5VMD1uPfcE-oWhIGvuiDqnwACkUj4zikLoQa1E9D5Opnz6XKovcPH_KO17jL6y6nioOIFphUdGAQ9Ogs1Ikncr1uA4kAarWbGkij4edu1FwKiQGuY2LO1idJwDWIRNkZwf5LKacsDEw1QnDAi_0TqR94ckcR8ZzQ4SxqxxVIM5TKqkkskyD620HMvL2zgX-bjNnMuLEEsWXEYvzluupu9qTemAXRHxg9i7WfUkqk4gI3YWbxp-2BwCfoimtxVzeaACfVMFOQrihSNKc9dacf7CQZeDnUGpLumteoCehJehE09pX2pM8P4dPnuiUPNXgYDZLBcvyc8ftSMJJ1q_fZ7R1TTgVXdVkt4O3CjFBVjeBbU-ngH6DLFmns-x11gw_vaAGyLSmLzp7oHQE0cpBM2-6yJglezpJTcQ5MTVUVDNfziVud-PZ8wK59zJJRPkDoxmq8FQmmoci-fuZWUEoQM1atW0PvjRJZOFNY2Bwo91TWGA7lIFdJxCiVD0ehMSFo9YV4JUuRuQpE0mHeM6sS-DYPc3_Pf2Cm9ll029iiCq6NTTbmYZ4lnKcFZE3VnVvL3X8NaG6kHz3yQ1TlGon0hkSUDg0ZEuTNpOhWQUHxGAuV4rXMKDSF3xj1dt3_bulKbx3v1op9ILbQYCda3XrPypKpJGFiwmAPo1FwHi0tjkFlLJh7IzLRRR2xplm_Mpe3Y5lT2C7ikJHP32aAY360PhujaDB18oF8ORHre3aG_ljlm-CBNod4BqX7n4loZacseJa4h_GE" rel="nofollow">build-log.txt</a> in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="134445638" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/21439" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/21439/hovercard" href="https://github.com/kubernetes/kubernetes/pull/21439">#21439</a> (before I retest) (Tried to find a flake dupe but didn't see an obvious one):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0217 18:27:49.485036   12269 manager.go:1620] Found pod infra container for &quot;phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)&quot;
I0217 18:27:49.485120   12269 manager.go:1633] Pod infra container looks good, keep it &quot;phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)&quot;
I0217 18:27:49.485169   12269 manager.go:1657] pod &quot;phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)&quot; container &quot;c1&quot; exists as /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a
I0217 18:27:49.485201   12269 manager.go:1725] Got container changes for pod &quot;phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)&quot;: {StartInfraContainer:false InfraChanged:false InfraContainerId:/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119 ContainersToStart:map[] ContainersToKeep:map[/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119:-1 /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a:0]}
I0217 18:27:49.692180   12269 integration.go:958] Scheduler doesn't make phantom pods: test passed.
I0217 18:27:49.692211   12269 integration.go:1068] 

Logging high latency metrics from the 10250 kubelet
I0217 18:27:49.716299   12269 server.go:1083] GET /metrics: (23.237036ms) 200 [[Go-http-client/1.1] 127.0.0.1:56949]
�������iI0217 18:27:49.725600   12269 integration.go:1070] 

Logging high latency metrics from the 10251 kubelet
Feb 17 18:27:49.725: INFO: 
Latency metrics for node localhost:10250
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:6.002896s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:5.002607s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.9 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1.001256s}
Error deleting container: Error response from daemon: Conflict, You cannot remove a running container. Stop the container before attempting removal or use -f
Unrecognized input header
Build step 'Execute shell' marked build as failure
Recording test results"><pre class="notranslate"><code class="notranslate">I0217 18:27:49.485036   12269 manager.go:1620] Found pod infra container for "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)"
I0217 18:27:49.485120   12269 manager.go:1633] Pod infra container looks good, keep it "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)"
I0217 18:27:49.485169   12269 manager.go:1657] pod "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)" container "c1" exists as /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a
I0217 18:27:49.485201   12269 manager.go:1725] Got container changes for pod "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)": {StartInfraContainer:false InfraChanged:false InfraContainerId:/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119 ContainersToStart:map[] ContainersToKeep:map[/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119:-1 /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a:0]}
I0217 18:27:49.692180   12269 integration.go:958] Scheduler doesn't make phantom pods: test passed.
I0217 18:27:49.692211   12269 integration.go:1068] 

Logging high latency metrics from the 10250 kubelet
I0217 18:27:49.716299   12269 server.go:1083] GET /metrics: (23.237036ms) 200 [[Go-http-client/1.1] 127.0.0.1:56949]
�������iI0217 18:27:49.725600   12269 integration.go:1070] 

Logging high latency metrics from the 10251 kubelet
Feb 17 18:27:49.725: INFO: 
Latency metrics for node localhost:10250
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:6.002896s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:5.002607s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.9 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1.001256s}
Error deleting container: Error response from daemon: Conflict, You cannot remove a running container. Stop the container before attempting removal or use -f
Unrecognized input header
Build step 'Execute shell' marked build as failure
Recording test results
</code></pre></div>