<p dir="auto"><a href="https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28599/kubernetes-pull-build-test-e2e-gce/58426/" rel="nofollow">https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28599/kubernetes-pull-build-test-e2e-gce/58426/</a></p>
<p dir="auto">Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:88
Expected error:
    &lt;*errors.errorString | 0xc820c80740&gt;: {
        s: &quot;error while stopping RC: rc-light-ctrl: timed out waiting for \&quot;rc-light-ctrl\&quot; to be synced&quot;,
    }
    error while stopping RC: rc-light-ctrl: timed out waiting for &quot;rc-light-ctrl&quot; to be synced
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:307"><pre class="notranslate"><code class="notranslate">/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:88
Expected error:
    &lt;*errors.errorString | 0xc820c80740&gt;: {
        s: "error while stopping RC: rc-light-ctrl: timed out waiting for \"rc-light-ctrl\" to be synced",
    }
    error while stopping RC: rc-light-ctrl: timed out waiting for "rc-light-ctrl" to be synced
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:307
</code></pre></div>
<p dir="auto">Happened on a presubmit run in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="164315792" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28599" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/28599/hovercard" href="https://github.com/kubernetes/kubernetes/pull/28599">#28599</a>.</p>
<p dir="auto">Previous issues for this test: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="160462664" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27443" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27443/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27443">#27443</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="161592369" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27835" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27835/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27835">#27835</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="165361685" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28900" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28900/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28900">#28900</a></p>
<p dir="auto">Actually there are 2 results in density test that seems conflict but NOT.</p>
<h3 dir="auto">1.throughput case</h3>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/1173940/11833686/36daafea-a400-11e5-8a5e-0b854e78c013.png"><img src="https://cloud.githubusercontent.com/assets/1173940/11833686/36daafea-a400-11e5-8a5e-0b854e78c013.png" alt="12_11_30" style="max-width: 100%;"></a></p>
<p dir="auto">when run the density test, this is the 1st test that shows count of different pod status. This pic indicate the throughput of apiserver+controller/schduler/kubelet. As the waiting curve drop down slowly and the pending curve is always 0. So that  scheduler is the main bottleneck  in the system when create a lot of pod at one time.</p>
<h3 dir="auto">2. one pod latency case</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Nov 27 19:10:28.552: INFO: 10% worst schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-13 33.1.1.212 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-214 33.1.1.228 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-251 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-276 33.1.1.193 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-174 33.1.1.89 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-134 33.1.1.214 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-90 33.1.2.20 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-105 33.1.1.159 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-140 33.1.2.71 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-63 33.1.1.192 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-125 33.1.1.96 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-144 33.1.1.104 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-54 33.1.1.172 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-118 33.1.2.37 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-23 33.1.1.201 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-75 33.1.2.78 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-99 33.1.1.98 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-253 33.1.1.244 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-137 33.1.2.56 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-18 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-254 33.1.2.32 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-168 33.1.2.29 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-146 33.1.2.51 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-142 33.1.2.40 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-145 33.1.2.37 2s}]
Nov 27 19:10:28.553: INFO: perc50: 0, perc90: 1s, perc99: 2s
Nov 27 19:10:28.553: INFO: 10% worst run-after-schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-20 33.1.2.65 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-269 33.1.2.39 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-250 33.1.1.126 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-206 33.1.1.85 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-151 33.1.1.120 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-217 33.1.1.187 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-46 33.1.1.227 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-11 33.1.1.165 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-182 33.1.1.120 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-157 33.1.1.133 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-5 33.1.1.177 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-3 33.1.1.143 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-31 33.1.1.149 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-95 33.1.1.124 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-69 33.1.1.189 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-220 33.1.1.142 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-184 33.1.1.168 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-74 33.1.1.112 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-1 33.1.1.154 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-247 33.1.1.165 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-178 33.1.1.139 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-93 33.1.1.159 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-127 33.1.1.160 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-164 33.1.1.182 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-2 33.1.1.196 5s}]
Nov 27 19:10:28.553: INFO: perc50: 2s, perc90: 3s, perc99: 5s"><pre class="notranslate"><code class="notranslate">Nov 27 19:10:28.552: INFO: 10% worst schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-13 33.1.1.212 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-214 33.1.1.228 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-251 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-276 33.1.1.193 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-174 33.1.1.89 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-134 33.1.1.214 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-90 33.1.2.20 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-105 33.1.1.159 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-140 33.1.2.71 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-63 33.1.1.192 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-125 33.1.1.96 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-144 33.1.1.104 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-54 33.1.1.172 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-118 33.1.2.37 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-23 33.1.1.201 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-75 33.1.2.78 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-99 33.1.1.98 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-253 33.1.1.244 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-137 33.1.2.56 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-18 33.1.2.26 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-254 33.1.2.32 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-168 33.1.2.29 1s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-146 33.1.2.51 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-142 33.1.2.40 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 2s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-145 33.1.2.37 2s}]
Nov 27 19:10:28.553: INFO: perc50: 0, perc90: 1s, perc99: 2s
Nov 27 19:10:28.553: INFO: 10% worst run-after-schedule latencies: [{density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-20 33.1.2.65 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-269 33.1.2.39 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-250 33.1.1.126 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-206 33.1.1.85 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-66 33.1.1.232 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-151 33.1.1.120 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-217 33.1.1.187 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-46 33.1.1.227 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-143 33.1.1.179 3s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-11 33.1.1.165 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-182 33.1.1.120 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-157 33.1.1.133 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-5 33.1.1.177 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-3 33.1.1.143 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-31 33.1.1.149 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-95 33.1.1.124 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-69 33.1.1.189 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-131 33.1.1.169 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-220 33.1.1.142 4s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-184 33.1.1.168 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-74 33.1.1.112 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-1 33.1.1.154 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-247 33.1.1.165 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-178 33.1.1.139 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-93 33.1.1.159 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-127 33.1.1.160 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-164 33.1.1.182 5s} {density-latency-pod-3d494294-953a-11e5-a6ec-fa163e92f3f5-2 33.1.1.196 5s}]
Nov 27 19:10:28.553: INFO: perc50: 2s, perc90: 3s, perc99: 5s
</code></pre></div>
<p dir="auto">Again in the density test. this case create one pod a time and observe its latency, which shows that scheduler latency is 2s and run-after-scheduler is 5s. Is that means scheduler is faster than other component is the system?</p>
<h3 dir="auto">3. explanation</h3>
<p dir="auto">the conflict between two case is not a real conflict. scheduler is  sequential across pod, but kubelet is naturally parallel.</p>
<p dir="auto">for scheduler,</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="latency * throughput = 1"><pre class="notranslate"><code class="notranslate">latency * throughput = 1
</code></pre></div>
<p dir="auto">as scheduler has just one loop to process all pods</p>
<p dir="auto">but for kubelet</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="latency * throughput = n(count of nodes)"><pre class="notranslate"><code class="notranslate">latency * throughput = n(count of nodes)
</code></pre></div>
<p dir="auto">maybe the latency of kubelet is near or slow than scheduler, but there are many kubelets run in system. If <code class="notranslate">latency(scheduler) == latency(kubelet)</code>, and there are 500 nodes, the <code class="notranslate">throughput(kubelet) == 500 times of throughput(scheduler)</code>. this explains the different result of these 2 tests.</p>
<p dir="auto"><strong>When cluster scale, the throughput gap between scheduler and kubelet/apiserver will be bigger.</strong><br>
<strong>So it is necessary to make scheduler parallel across pod.</strong></p>
<h3 dir="auto">environment</h3>
<ul dir="auto">
<li>20 phycial machine 40 core, 380G, with openstack</li>
<li>k8s-master(1VM)     8 core, 16 G</li>
<li>k8s-node(500VM)    2 core, 4G</li>
<li>etcd(5VM)                4 core, 8G</li>
<li>ubuntu 14.04</li>
<li>Go 1.4.3</li>
<li>Etcd 2.2.2</li>
<li>K8s master head of 12.7</li>
</ul>