<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.5&quot;, GitCommit:&quot;b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-11T20:29:08Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-01T19:19:19Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:29:08Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:19:19Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): The one created by <code class="notranslate">kube-up.sh</code></li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): The one used by <code class="notranslate">kube-up.sh</code></li>
<li><strong>Install tools</strong>: kube-up.sh</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl  --kubeconfig=infra/dev/kubeconfig drain --ignore-daemonsets --force --delete-local-data --grace-period=120 ip-172-20-0-123.ec2.internal 
node &quot;ip-172-20-0-123.ec2.internal&quot; already cordoned
error: replicationcontrollers &quot;memcached&quot; not found: memcached-4n9cy, memcached-4n9cy"><pre class="notranslate"><code class="notranslate">kubectl  --kubeconfig=infra/dev/kubeconfig drain --ignore-daemonsets --force --delete-local-data --grace-period=120 ip-172-20-0-123.ec2.internal 
node "ip-172-20-0-123.ec2.internal" already cordoned
error: replicationcontrollers "memcached" not found: memcached-4n9cy, memcached-4n9cy
</code></pre></div>
<p dir="auto">It would seem that <code class="notranslate">memcached-4n9cy</code> is an orphan pod from a replication controller that doesn't exist anymore.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ ./kubectl  --kubeconfig=infra/dev/kubeconfig describe pod memcached-4n9cy | grep Controllers
Controllers:    ReplicationController/memcached"><pre class="notranslate"><code class="notranslate">$ ./kubectl  --kubeconfig=infra/dev/kubeconfig describe pod memcached-4n9cy | grep Controllers
Controllers:    ReplicationController/memcached
</code></pre></div>
<p dir="auto">We moved to ReplicaSets and don't have any ReplicationControllers left:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ ./kubectl  --kubeconfig=infra/dev/kubeconfig get --all-namespaces rc 
NAMESPACE     NAME                             DESIRED   CURRENT   AGE
kube-system   kube-dns-v17                     1         1         41d
kube-system   kubernetes-dashboard-v1.1.0      1         1         41d
kube-system   monitoring-influxdb-grafana-v3   1         1         41d"><pre class="notranslate"><code class="notranslate">$ ./kubectl  --kubeconfig=infra/dev/kubeconfig get --all-namespaces rc 
NAMESPACE     NAME                             DESIRED   CURRENT   AGE
kube-system   kube-dns-v17                     1         1         41d
kube-system   kubernetes-dashboard-v1.1.0      1         1         41d
kube-system   monitoring-influxdb-grafana-v3   1         1         41d
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Providing <code class="notranslate">--force</code> should allow you to drain the pods even if they are orphans from a replication controller.</p>
<p dir="auto">I had to manually remove the pod before running <code class="notranslate">kubectl drain</code> again to make it work.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">I am not sure how the pods ended up being orphaned but I guess that getting an orphaned pod from a replication controller should make it easy to reproduce.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">We had a micro-outage the other day when one of the machines in our cluster stopped trying to restart. On startup, it hit a transient error that prevented the server from starting up, but the pod did not try to restart again after that. The logs showed nothing between when it got stuck starting up and when I initiated a manual restart 20 minutes later. It seems we hit this issue during a network partition, though we are still investigating.</p>
<p dir="auto">Does the pod need to talk to the master in order for a restart to be triggered?</p>
<p dir="auto">Attached is our configuration file, including the restart policy <a href="https://github.com/kubernetes/kubernetes/files/192511/apiserverProdConfig.txt">apiserverProdConfig.txt</a></p>
<p dir="auto">Thanks!</p>
<p dir="auto">We're using Kubernetes 1.2.0</p>