<p dir="auto">I am opening this master/umbrella issue to track Ubernetes e2e test flakes. I think this issue is sufficient right now. But as we make progress we might have to fork this into sub-issues.</p>
<p dir="auto">This issue is not just about the e2e tests themselves but also includes infrastructure related flakes such as bringing up the clusters, bringing up the federation control plane, etc.</p>
<p dir="auto">Here is one flake that I am seeing from yesterday:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ FEDERATION=true E2E_ZONES=&quot;us-central1-f&quot; KUBE_GCE_ZONE=&quot;us-central1-f&quot; go run hack/e2e.go -v --up
...
...
...
NAME                                     LOCATION       SCOPE  BASE_INSTANCE_NAME                       SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                           AUTOSCALED
madhusudancs-us-central1-f-minion-group  us-central1-f  zone   madhusudancs-us-central1-f-minion-group        3            madhusudancs-us-central1-f-minion-template
Waiting for group to become stable, current operations: creating: 3
Waiting for group to become stable, current operations: creating: 3
Waiting for group to become stable, current operations: creating: 3
Waiting for group to become stable, current operations: creating: 3
Group is stable
INSTANCE_GROUPS=madhusudancs-us-central1-f-minion-group
NODE_NAMES=madhusudancs-us-central1-f-minion-group-5wk9 madhusudancs-us-central1-f-minion-group-6ys7 madhusudancs-us-central1-f-minion-group-l1vs
Using master: madhusudancs-us-central1-f-master (external IP: &lt;REDACTED_IP&gt;)
Waiting up to 300 seconds for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This may time out if there was some uncaught error during start up.

...................................................................................................................................................Cluster failed to initialize within 300 seconds.
2016/06/02 14:18:06 e2e.go:218: Error running up: exit status 1
2016/06/02 14:18:06 e2e.go:214: Step 'up' finished in 8m0.261564635s
2016/06/02 14:18:06 e2e.go:114: Error starting e2e cluster. Aborting.
exit status 1"><pre class="notranslate">$ FEDERATION=true E2E_ZONES=<span class="pl-s"><span class="pl-pds">"</span>us-central1-f<span class="pl-pds">"</span></span> KUBE_GCE_ZONE=<span class="pl-s"><span class="pl-pds">"</span>us-central1-f<span class="pl-pds">"</span></span> go run hack/e2e.go -v --up
...
...
...
NAME                                     LOCATION       SCOPE  BASE_INSTANCE_NAME                       SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                           AUTOSCALED
madhusudancs-us-central1-f-minion-group  us-central1-f  zone   madhusudancs-us-central1-f-minion-group        3            madhusudancs-us-central1-f-minion-template
Waiting <span class="pl-k">for</span> group to become stable, current operations: creating: 3
Waiting <span class="pl-k">for</span> group to become stable, current operations: creating: 3
Waiting <span class="pl-k">for</span> group to become stable, current operations: creating: 3
Waiting <span class="pl-k">for</span> group to become stable, current operations: creating: 3
Group is stable
INSTANCE_GROUPS=madhusudancs-us-central1-f-minion-group
NODE_NAMES=madhusudancs-us-central1-f-minion-group-5wk9 madhusudancs-us-central1-f-minion-group-6ys7 madhusudancs-us-central1-f-minion-group-l1vs
Using master: madhusudancs-us-central1-f-master (external IP: <span class="pl-k">&lt;</span>REDACTED_IP<span class="pl-k">&gt;</span>)
Waiting up to 300 seconds <span class="pl-k">for</span> cluster initialization.

  This will continually check to see <span class="pl-k">if</span> the API <span class="pl-k">for</span> kubernetes is reachable.
  This may <span class="pl-k">time</span> out <span class="pl-k">if</span> there was some uncaught error during start up.

...................................................................................................................................................Cluster failed to initialize within 300 seconds.
2016/06/02 14:18:06 e2e.go:218: Error running up: <span class="pl-c1">exit</span> status 1
2016/06/02 14:18:06 e2e.go:214: Step <span class="pl-s"><span class="pl-pds">'</span>up<span class="pl-pds">'</span></span> finished <span class="pl-k">in</span> 8m0.261564635s
2016/06/02 14:18:06 e2e.go:114: Error starting e2e cluster. Aborting.
<span class="pl-c1">exit</span> status 1</pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/colhom/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/colhom">@colhom</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikhiljindal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikhiljindal">@nikhiljindal</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/colhom/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/colhom">@colhom</a> @kubernetes/sig-cluster-federation <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jianhuiz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jianhuiz">@jianhuiz</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mfanjie/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mfanjie">@mfanjie</a></p>
<p dir="auto">We reverted it to use Update (PUT) while scale was broken. We should revert back, for ReplicationController, ReplicaSet, and Deployment.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/janetkuo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/janetkuo">@janetkuo</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/madhusudancs/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/madhusudancs">@madhusudancs</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/soltysh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/soltysh">@soltysh</a> @kargakis</p>