<p dir="auto">There is quite a number of duplicated code in different node status related test functions, such as mock cadvisor intialization, expected node status etc.</p>
<p dir="auto">It should be cleaned up. :)</p>
<p dir="auto">Admittedly this one is an edge case, but we had an issue that nearly locked us out of our AWS account following data loss on the kubernetes master. Specifically:</p>
<ol dir="auto">
<li>We turned up a k8s cluster integrated with AWS</li>
<li>An issue in our infrastructure caused us to lose the master's etcd storage (since resolved)</li>
<li>The new master came up, we recreated some of the lost state (i.e. kubectl create -f service.yml) and the master tried to resync that partial state with AWS, failing on duplicate resources</li>
<li>Until those resources were manually removed, the controller retried each operation often enough that our account become API rate limited, degrading access to our infrastructure</li>
</ol>
<p dir="auto">The obvious solution is to not lose the master's data, and we have fixed our mistake there. That said, as the flow above nearly locked us out of our AWS account, it would be nice if the controller had responded a little better to the partial data scenario. Two changes that would have helped us:</p>
<ol dir="auto">
<li>Increasing backoffs (random, exponential, fibonacci, etc.) around AWS API retries especially when the previous call failed due to a "RequestLimitExceeded" – though this does have the downside of pushing out time to convergence when the underlying issue is fixed</li>
<li>A way for the AWS integration to self-heal: when it fails to create e.g. a route because that route already exists in the table, a nicer error handling strategy might simply say "Creation successful!" – the situation is more difficult with security groups that have some state that potentially requires diffing, but in our case the issue occurred in the degenerate case where the controller was trying to recreate the exact resource that already existed</li>
</ol>
<p dir="auto">For context, the log messages we saw from the controller (there was some variation depending on which exact request was throttled):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E0916 23:21:30.178529       1 servicecontroller.go:171] Failed to process service delta. Retrying: failed to create external load balancer for service default/hello-world: error listing AWS subnets: RequestLimitExceeded: Request limit exceeded.
    status code: 503, request id: []"><pre class="notranslate"><code class="notranslate">E0916 23:21:30.178529       1 servicecontroller.go:171] Failed to process service delta. Retrying: failed to create external load balancer for service default/hello-world: error listing AWS subnets: RequestLimitExceeded: Request limit exceeded.
    status code: 503, request id: []
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E0916 22:53:56.348372       1 routecontroller.go:91] Could not create route ROUTE_ID 10.244.0.0/24: error creating AWS route (10.244.0.0/24): RequestLimitExceeded: Request limit exceeded."><pre class="notranslate"><code class="notranslate">E0916 22:53:56.348372       1 routecontroller.go:91] Could not create route ROUTE_ID 10.244.0.0/24: error creating AWS route (10.244.0.0/24): RequestLimitExceeded: Request limit exceeded.
</code></pre></div>
<p dir="auto">The underlying issues seemed to stem from:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E0916 23:35:37.607891       1 servicecontroller.go:171] Failed to process service delta. Retrying: failed to create external load balancer for service default/hello-world: InvalidGroup.Duplicate: The security group 'SECURITY_GROUP_ID' already exists for VPC 'VPC_ID'"><pre class="notranslate"><code class="notranslate">E0916 23:35:37.607891       1 servicecontroller.go:171] Failed to process service delta. Retrying: failed to create external load balancer for service default/hello-world: InvalidGroup.Duplicate: The security group 'SECURITY_GROUP_ID' already exists for VPC 'VPC_ID'
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E0916 22:52:56.239869       1 routecontroller.go:91] Could not create route ROUTE_ID 10.244.3.0/24: error creating AWS route (10.244.3.0/24): RouteAlreadyExists: The route identified by 10.244.3.0/24 already exists."><pre class="notranslate"><code class="notranslate">E0916 22:52:56.239869       1 routecontroller.go:91] Could not create route ROUTE_ID 10.244.3.0/24: error creating AWS route (10.244.3.0/24): RouteAlreadyExists: The route identified by 10.244.3.0/24 already exists.
</code></pre></div>
<p dir="auto">Deleting both of those resources did appear to resolve the problem. Thanks for reading my book!</p>