<p dir="auto">We had a quite serious case today which caused all our pods to terminate because AWS had trouble with their EC2 service in Ireland (6:00 AM PDT)</p>
<p dir="auto">Basically after 40s (corresponding to <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/admin/kube-controller-manager.md"><code class="notranslate">node-monitor-grace-period</code></a>) of API downtime during which kubelet couldn't list EC2 instances, it hit the maximum node status retry count and killed everything:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="12:44:22 kubelet.go:1933] Error updating node status, will retry: failed to get node address from cloud provider: error listing AWS instances: Unavailable: The service is unavailable. Please try again shortly.
status code: 503, request id: []

...

12:45:04 kubelet.go:839] Unable to update node status: update node status exceeds retry count
12:45:10 kubelet.go:1518] Killing unwanted pod &quot;swagger-extcs&quot;
12:45:10 manager.go:1123] Killing container with id &quot;aa4a55984...&quot;
12:45:10 manager.go:1123] Killing container with id &quot;9bdb1788e...&quot;
12:45:10 kubelet.go:1343] Orphaned volume &quot;5796b5eb...&quot; found, tearing down volume
..."><pre class="notranslate"><code class="notranslate">12:44:22 kubelet.go:1933] Error updating node status, will retry: failed to get node address from cloud provider: error listing AWS instances: Unavailable: The service is unavailable. Please try again shortly.
status code: 503, request id: []

...

12:45:04 kubelet.go:839] Unable to update node status: update node status exceeds retry count
12:45:10 kubelet.go:1518] Killing unwanted pod "swagger-extcs"
12:45:10 manager.go:1123] Killing container with id "aa4a55984..."
12:45:10 manager.go:1123] Killing container with id "9bdb1788e..."
12:45:10 kubelet.go:1343] Orphaned volume "5796b5eb..." found, tearing down volume
...
</code></pre></div>
<p dir="auto">I could certainly assume that downtimes at AWS may never last more than <code class="notranslate">t</code> minutes and increase the <code class="notranslate">node-monitor-grace-period</code> and <code class="notranslate">pod-eviction-timeout</code> in my kube-controller-manager accordingly, but there is something dangerous in making such assumptions and relying <em>only</em> on the cloud provider to define nodes' health, since a not responding cloud provider doesn't mean a completely unhealthy platform.</p>
<p dir="auto">We paid that price today and saw our whole prod cluster going down while everything was fine on the instances themselves 🙈</p>
<p dir="auto">The way we access the kubernetes service is inconsistent; in all other namespaces it is <code class="notranslate">https://kubernetes</code>, but in kube-system I don't have a <code class="notranslate">kubernetes</code> service (should I?).</p>
<p dir="auto">When I try to use <code class="notranslate">kubernetes</code> from go I get this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Get https://kubernetes/api: dial tcp: lookup kubernetes on 100.64.0.10:53: server misbehaving"><pre class="notranslate"><code class="notranslate">Get https://kubernetes/api: dial tcp: lookup kubernetes on 100.64.0.10:53: server misbehaving
</code></pre></div>
<p dir="auto">bboreham pointed out that I can use <code class="notranslate">KUBERNETES_SERVICE_HOST</code>, but I feel this inconsistently is likely to continue to surprise people.</p>
<p dir="auto">It may be something wrong in my configuration (i.e. perhaps the kubernetes service should already be registered in kube-system), but is there some reason <em>not</em> to register it?</p>