<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):  yes, but there seems to be a bug</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<h2 dir="auto">kubernetes delete daemonset fails</h2>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.4&quot;, GitCommit:&quot;dd6b458ef8dbf24aff55795baa68f83383c9b3a9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-01T16:45:16Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.4&quot;, GitCommit:&quot;dd6b458ef8dbf24aff55795baa68f83383c9b3a9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-01T16:38:31Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux ip-172-20-105-42 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: kops</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I have ingress running as a daemonset, looking for a nodeSelector:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="      nodeSelector:
        roles.kubernetes.io/node: &quot;1&quot;"><pre class="notranslate"><code class="notranslate">      nodeSelector:
        roles.kubernetes.io/node: "1"
</code></pre></div>
<p dir="auto">I am updating it to have a TCP services config map, so I needed to delete the daemonset first.<br>
Ran: <code class="notranslate">kubectl delete ds nginx-ingress-controller --namespace=kube-system</code><br>
And waited.</p>
<p dir="auto">Eventually, it failed with <code class="notranslate">error: timed out waiting for the condition</code></p>
<p dir="auto">But <code class="notranslate">kubectl get pods --namespace=kube-system -w</code> shows that the pods are being terminated, and then being recreated.</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
All pods and the daemonset should be deleted.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
I have no idea.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
Not sure.</p>
<p dir="auto">We have a bunch of different controllers already and (at least now) they are running on a single binary.</p>
<p dir="auto">While looking into logs (in apiserver) one can see that a bunch of them are doing very similar operations at the same time, e.g.:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I1002 11:46:31.421115      12 handlers.go:130] GET /api/v1/pods: (1.325473918s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48235]
I1002 11:46:31.425218      12 handlers.go:130] GET /api/v1/pods: (1.329777158s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48234]
I1002 11:46:31.456389      12 handlers.go:130] GET /api/v1/pods: (1.360542647s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48236]"><pre class="notranslate"><code class="notranslate">I1002 11:46:31.421115      12 handlers.go:130] GET /api/v1/pods: (1.325473918s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48235]
I1002 11:46:31.425218      12 handlers.go:130] GET /api/v1/pods: (1.329777158s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48234]
I1002 11:46:31.456389      12 handlers.go:130] GET /api/v1/pods: (1.360542647s) 200 [[kube-controller-manager/v1.1.0 (linux/amd64) kubernetes/521c67a] 127.0.0.1:48236]
</code></pre></div>
<p dir="auto">In the above example, there are 3 "list pods" opertion send at the same time (those are comming from EndpointController, JobController and ReplicationController).<br>
This seems to be a waster of resource (both cpu, memory and network).</p>
<p dir="auto">We should try to have one "store" per resource and share it between different controllers (that will be able to register their callbacks on that store).<br>
Thoughts?</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a> @kubernetes/goog-csi <a class="team-mention js-team-mention notranslate" data-error-text="Failed to load team members" data-id="3959732" data-permission-text="Team members are private" data-url="/orgs/kubernetes/teams/sig-scalability/members" data-hovercard-type="team" data-hovercard-url="/orgs/kubernetes/teams/sig-scalability/hovercard" href="https://github.com/orgs/kubernetes/teams/sig-scalability">@kubernetes/sig-scalability</a></p>