<p dir="auto">On <code class="notranslate">2012.12.18</code> the AWS region us-east-1 experienced an outage affecting service of the DescribeInstances and DescribeTags APIs. This is turn brought our two independent clusters operating in the region, as <a href="https://github.com/kubernetes/kubernetes/blob/7069a811164f68473dd1b94ee2b2abc169e35d81/pkg/cloudprovider/providers/aws/aws.go#L2100-L2119">the Kubelet AWS clustering mechanisms</a> rely on the DescribeInstances API. Networking was unaffected - master and nodes could still communicate without problem.</p>
<p dir="auto">I propose that this AWS failure should not bring down a cluster.</p>
<p dir="auto">The relevant Kubelet log message:</p>
<blockquote>
<p dir="auto">Error updating node status, will retry: failed to get node address from cloud provider: no instances found for name: ip-<em>-</em>-<em>-</em>.ec2.internal</p>
</blockquote>
<p dir="auto">There are several issues that relate to this functionality. I am filing this issue to ensure that Kubernetes is not needlessly vulnerable to this type of outage.</p>
<p dir="auto"><em>Possibly Related</em><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="118356679" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/17643" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/17643/hovercard" href="https://github.com/kubernetes/kubernetes/pull/17643">#17643</a> use private ip instead of local hostname for AWS<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="121563578" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/18525" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/18525/hovercard" href="https://github.com/kubernetes/kubernetes/issues/18525">#18525</a> We shouldn't use node names as addresses</p>
<p dir="auto">Incident details from <a href="http://status.aws.amazon.com/" rel="nofollow">http://status.aws.amazon.com/</a></p>
<blockquote>
<p dir="auto">1:02 PM PST We are investigating elevated latencies and error rates for the DescribeInstances and DescribeTags APIs in the us-east-1 region. Customers may observe elevated error rates or missing tags for instances in the AWS Management Console.<br>
1:44 PM PST We have identified the root cause of elevated latencies and error rates for the DescribeInstances and DescribeTags APIs and are currently working to resolve. Customers may observe elevated error rates or missing tags for instances in the AWS Management Console.<br>
3:03 PM PST Between 12:30 PM and 2:44 PM PST, we experienced elevated latencies and error rates for the DescribeInstances and DescribeTags APIs. During this period, customers may have observed elevated error rates and missing tags for instances in the AWS Management Console. During this event, instances continued running normally. Tags requested during this period are now visible. This issue has been resolved and the service is operating normally.</p>
</blockquote>
<p dir="auto">Generate a UUID when you kube-up.sh a cluster.</p>
<p dir="auto">One use case is for associating credentials with multiple clusters.</p>
<ul dir="auto">
<li>.kubernetes_auth format could have sections for each cluster UUID.  So, if you start an e2e cluster it won't clobber your creds for your dev cluster.  A flag to kubectl could let you set what UUID you are talking to.
<ul dir="auto">
<li>e.g.:  <code class="notranslate">{"UUID": "123-...", "BearerToken": "abc"}\n { "UUID": "456-...", "User": "foo", "Password":"bar"}</code></li>
</ul>
</li>
<li>more generally, if you have clusters in several regions.  You don't want to identify by master IP since that might change.  And you don't want to rely on DNS because you don't always have a DNS mapping from cluster-name to master IPs.</li>
<li>when we want to name resources that are unique to a cluster using an URL that is portable across clusters.
<ul dir="auto">
<li>E.g:  <code class="notranslate">kubernetes:123-456-789-234234-567567/api/v1beta1/ns/mynamespace/pods/mypodname</code></li>
</ul>
</li>
</ul>
<p dir="auto">Garbage collection of old UUIDs in the <code class="notranslate">.kubernetes_auth</code> is left as an exercise to the user.</p>