<p dir="auto">Two nodes went away as I updated the instances, but they still appear and, more importantly/confusingly, their daemonset pods are still in Running state.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ fleetctl list-machines
MACHINE         IP              METADATA
729947b9...     10.0.12.180     role=node
a0883347...     10.0.8.4        instance=m0,role=master
$ kubectl get nodes
NAME                          STATUS                     AGE
ip-10-0-10-222.ec2.internal   NotReady                   2d
ip-10-0-12-147.ec2.internal   NotReady                   1d
ip-10-0-12-180.ec2.internal   Ready                      17m
ip-10-0-8-4.ec2.internal      Ready,SchedulingDisabled   2d
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                    READY     STATUS        RESTARTS   AGE
kube-system   fluentd-logging-2gxb1   1/1       Running       0          1d
kube-system   fluentd-logging-33y2f   1/1       Running       0          1d
kube-system   fluentd-logging-b1yuv   1/1       Running       0          17m
kube-system   fluentd-logging-o9hvp   1/1       Running       0          1d"><pre class="notranslate"><code class="notranslate">$ fleetctl list-machines
MACHINE         IP              METADATA
729947b9...     10.0.12.180     role=node
a0883347...     10.0.8.4        instance=m0,role=master
$ kubectl get nodes
NAME                          STATUS                     AGE
ip-10-0-10-222.ec2.internal   NotReady                   2d
ip-10-0-12-147.ec2.internal   NotReady                   1d
ip-10-0-12-180.ec2.internal   Ready                      17m
ip-10-0-8-4.ec2.internal      Ready,SchedulingDisabled   2d
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                    READY     STATUS        RESTARTS   AGE
kube-system   fluentd-logging-2gxb1   1/1       Running       0          1d
kube-system   fluentd-logging-33y2f   1/1       Running       0          1d
kube-system   fluentd-logging-b1yuv   1/1       Running       0          17m
kube-system   fluentd-logging-o9hvp   1/1       Running       0          1d
</code></pre></div>
<p dir="auto">There are only two instances and thus two daemons up, not four.<br>
I understand the rationale for not killing the pods outright, but is there a timeout after which they change state or perhaps the nodes do?</p>
<p dir="auto">My concern is twofold:</p>
<ol dir="auto">
<li><code class="notranslate">Running</code> is a lie. Things will be confusing for monitoring/accounting/troubleshooting purposes.</li>
<li>State about pods and nodes is kept around for a long time. Imagine someone using DaemonSets on a pool of EC2 spot instances...</li>
</ol>
<p dir="auto">I don't see any existing issues filed for this.</p>
<p dir="auto">Allow the user to instruct Kubernetes,  based on data from liveness/readiness probes (or failures to start new pods at all), to automatically undo a rollout, not just pause it. This could even have a configurable maxFailures knob to specify how many pods in "failed after upgrade" state to allow before rolling back. Similar for pending pods.</p>