<p dir="auto">Over a year ago, we had the <code class="notranslate">kubernetes-build</code> job down &lt;5m. It's now coming in at 30m. Some of that shot up really recently, too, looking at the build time trend:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/4942464/16088152/95720012-32da-11e6-981c-410fa6a2abb1.png"><img src="https://cloud.githubusercontent.com/assets/4942464/16088152/95720012-32da-11e6-981c-410fa6a2abb1.png" alt="time-trend" style="max-width: 100%;"></a></p>
<p dir="auto">It looks like sometime around <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91142020" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/10384" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/10384/hovercard" href="https://github.com/kubernetes/kubernetes/issues/10384">#10384</a>, we shot up from a fat 20m another 50% to 30m. Can we at least eliminate that jump?</p>
<p dir="auto">Is there anything else we can do here? Maybe split the non-<code class="notranslate">amd64</code> builds off to a separate job so that our whole CI infrastructure isn't tied to them?</p>
<p dir="auto"><a class="team-mention js-team-mention notranslate" data-error-text="Failed to load team members" data-id="2009229" data-permission-text="Team members are private" data-url="/orgs/kubernetes/teams/test-infra-maintainers/members" data-hovercard-type="team" data-hovercard-url="/orgs/kubernetes/teams/test-infra-maintainers/hovercard" href="https://github.com/orgs/kubernetes/teams/test-infra-maintainers">@kubernetes/test-infra-maintainers</a> @kubernetes/release-maintainers</p>
<p dir="auto">Version-Release number of selected component (if applicable):<br>
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.0.1310+8741217179860e", GitCommit:"8741217179860e9f0ce37997c810f61170a3672a", GitTreeState:"clean", BuildDate:"2016-07-07T06:54:04Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto">Steps to Reproduce:<br>
1.Install kubernetes with one master and two nodes<br>
2.Create a pod using replicationcontroller<br>
apiVersion: v1<br>
kind: ReplicationController<br>
metadata:<br>
name: chaoyangwildfly-rc<br>
labels:<br>
name: chaoyangwildfly<br>
context: docker-k8s-lab<br>
spec:<br>
replicas: 1<br>
template:<br>
metadata:<br>
labels:<br>
name: chaoyangwildfly<br>
spec:<br>
containers:<br>
- name: chaoyangwildfly-rc-pod<br>
image: jhou/hello-openshift<br>
ports:<br>
- containerPort: 8080<br>
volumeMounts:<br>
- name: html-volume<br>
mountPath: "/usr/share/nginx/html"<br>
volumes:<br>
- name: html-volume<br>
awsElasticBlockStore:<br>
volumeID: aws://us-east-1d/vol-dde44879<br>
fsType: ext4<br>
3.Check pod status<br>
[root@ip-172-18-5-143 ~]# kubectl describe pods chaoyangwildfly-rc-ccx51<br>
Name:        chaoyangwildfly-rc-ccx51<br>
Namespace:    default<br>
Node:        ip-172-18-0-61.ec2.internal/172.18.0.61<br>
Start Time:    Fri, 08 Jul 2016 02:56:00 -0400<br>
Labels:        name=chaoyangwildfly<br>
Status:        Running<br>
IP:        172.16.73.10<br>
Controllers:    ReplicationController/chaoyangwildfly-rc<br>
Containers:<br>
chaoyangwildfly-rc-pod:<br>
Container ID:    docker://0bb522c3d8305c13decef15c5624637bb3824021be78ae75b086be1560369817<br>
Image:        jhou/hello-openshift<br>
Image ID:        docker://sha256:3642a95271f490f9d618e29128a089ebaaf58f8f3f4e556c02660b54ebb881fd<br>
Port:        8080/TCP<br>
QoS Tier:<br>
memory:            BestEffort<br>
cpu:            BestEffort<br>
State:            Running<br>
Started:            Fri, 08 Jul 2016 02:56:08 -0400<br>
Ready:            True<br>
Restart Count:        0<br>
Environment Variables:    <br>
Conditions:<br>
Type        Status<br>
Initialized     True<br>
Ready     False<br>
PodScheduled     True<br>
Volumes:<br>
html-volume:<br>
Type:    AWSElasticBlockStore (a Persistent Disk resource in AWS)<br>
VolumeID:    aws://us-east-1d/vol-dde44879<br>
FSType:    ext4<br>
Partition:    0<br>
ReadOnly:    false<br>
default-token-vs6y1:<br>
Type:    Secret (a volume populated by a Secret)<br>
SecretName:    default-token-vs6y1<br>
Events:<br>
FirstSeen    LastSeen    Count    From                    SubobjectPath                Type        Reason        Message</p>
<hr>
<p dir="auto">5m        5m        1    {default-scheduler }                                Normal        Scheduled    Successfully assigned chaoyangwildfly-rc-ccx51 to ip-172-18-0-61.ec2.internal<br>
5m        5m        1    {kubelet ip-172-18-0-61.ec2.internal}    spec.containers{chaoyangwildfly-rc-pod}    Normal        Pulling        pulling image "jhou/hello-openshift"<br>
5m        5m        1    {kubelet ip-172-18-0-61.ec2.internal}    spec.containers{chaoyangwildfly-rc-pod}    Normal        Pulled        Successfully pulled image "jhou/hello-openshift"<br>
5m        5m        1    {kubelet ip-172-18-0-61.ec2.internal}    spec.containers{chaoyangwildfly-rc-pod}    Normal        Created        Created container with docker id 0bb522c3d830<br>
5m        5m        1    {kubelet ip-172-18-0-61.ec2.internal}    spec.containers{chaoyangwildfly-rc-pod}    Normal        Started        Started container with docker id 0bb522c3d830</p>
<p dir="auto">4..After the pod is running, stop kubelet service on the node ip-172-18-0-61.ec2.internal<br>
5. A new pod will be assigned to the other node<br>
[root@ip-172-18-5-143 ~]# kubectl describe pods chaoyangwildfly-rc-hn9kn<br>
Name:        chaoyangwildfly-rc-hn9kn<br>
Namespace:    default<br>
Node:        ip-172-18-9-229.ec2.internal/172.18.9.229<br>
Start Time:    Fri, 08 Jul 2016 03:02:31 -0400<br>
Labels:        name=chaoyangwildfly<br>
Status:        Pending<br>
IP:<br>
Controllers:    ReplicationController/chaoyangwildfly-rc<br>
Containers:<br>
chaoyangwildfly-rc-pod:<br>
Container ID:<br>
Image:        jhou/hello-openshift<br>
Image ID:<br>
Port:        8080/TCP<br>
QoS Tier:<br>
cpu:            BestEffort<br>
memory:            BestEffort<br>
State:            Waiting<br>
Reason:            ContainerCreating<br>
Ready:            False<br>
Restart Count:        0<br>
Environment Variables:    <br>
Conditions:<br>
Type        Status<br>
Initialized     True<br>
Ready     False<br>
PodScheduled     True<br>
Volumes:<br>
html-volume:<br>
Type:    AWSElasticBlockStore (a Persistent Disk resource in AWS)<br>
VolumeID:    aws://us-east-1d/vol-dde44879<br>
FSType:    ext4<br>
Partition:    0<br>
ReadOnly:    false<br>
default-token-vs6y1:<br>
Type:    Secret (a volume populated by a Secret)<br>
SecretName:    default-token-vs6y1<br>
Events:<br>
FirstSeen    LastSeen    Count    From                    SubobjectPath    Type        Reason        Message</p>
<hr>
<p dir="auto">4m        4m        1    {default-scheduler }                    Normal        Scheduled    Successfully assigned chaoyangwildfly-rc-hn9kn to ip-172-18-9-229.ec2.internal<br>
2m        1s        2    {kubelet ip-172-18-9-229.ec2.internal}            Warning        FailedMount    Unable to mount volumes for pod "chaoyangwildfly-rc-hn9kn_default(f059eb80-44d9-11e6-ba51-0ecfeba772c9)": timeout expired waiting for volumes to attach/mount for pod "chaoyangwildfly-rc-hn9kn"/"default". list of unattached/unmounted volumes=[html-volume]<br>
2m        1s        2    {kubelet ip-172-18-9-229.ec2.internal}            Warning        FailedSync    Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "chaoyangwildfly-rc-hn9kn"/"default". list of unattached/unmounted volumes=[html-volume]</p>
<ol dir="auto">
<li>From aws web console, I foud volume vol-dde44879 is still attched to node ip-172-18-0-61.ec2.internal</li>
</ol>