<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No<br>
<strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
Features Request</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
[root@localhost ~]# kubectl version<br>
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1877+3f4a66f3d6892b-dirty", GitCommit:"3f4a66f3d6892b8d8831a8a60b91fd1afbefee4d", GitTreeState:"dirty", BuildDate:"2016-10-31T20:47:46Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto">[root@localhost ~]# kubeadm version<br>
kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:x89_64</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):Ubuntu 16.04</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux kubenode01 4.4.0-38-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35453339" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/57" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/57/hovercard" href="https://github.com/kubernetes/kubernetes/pull/57">#57</a>-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>:kubeadm</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">While running performance tests (start 30 pods per node in 50 nodes cluster) I faced the following problem.</p>
<ol dir="auto">
<li>Create replication controller and resize it to 1500. I saw the following logs:<br>
I0327 14:45:09.036772   14025 density.go:109] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 0 pods out of 1500<br>
I0327 14:45:16.918259   14025 density.go:109] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 835 pods out of 1500<br>
I0327 14:45:24.465116   14025 density.go:129] Controller my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b: Found 1500 pods out of 1500</li>
<li>However, looking into scheduler logs, the first time when it attempts to schedule any of those pods is related to the log:<br>
I0327 13:45:55.474005    4551 factory.go:166] About to try and schedule pod my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39bfjo6e</li>
</ol>
<p dir="auto">This is roughly ~45 seconds after the first pod was created. The question is what the scheduler was doing in the meantime - the only logs from that time are:<br>
I0327 13:40:52.230407    4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF<br>
E0327 13:40:52.231778    4551 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [4053/16]) [<br>
5052]<br>
I0327 13:45:53.272954    4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF<br>
I0327 13:45:53.273448    4551 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF<br>
E0327 13:45:53.277590    4551 reflector.go:149] watch of *api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [12989/5082]) [1<br>
3988]<br>
E0327 13:45:53.277823    4551 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [12989/5082]<br>
) [13988]<br>
I0327 13:45:55.474005    4551 factory.go:166] About to try and schedule pod my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39bfjo6e</p>
<p dir="auto">A very similar thing happened in Kubelet. During the same test, the first pod was bind to minion "*0jxm" at:<br>
I0327 13:46:00.291905    4551 factory.go:303] Attempting to bind my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b7aukl to e2e-test-wojtekt-minion-0jxm.c.groovy-sentry-504.internal</p>
<p dir="auto">However, Kubelet reacted to it 50 seconds later:<br>
I0327 13:46:50.560684    4712 kubelet.go:1099] No Infra Container for "my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b7aukl_default" found. All containers will be restarted.</p>
<p dir="auto">Again, it seems that nothing was really happening during that 50 seconds. The Kubelet logs from that time are:<br>
I0327 13:44:55.038542    4712 server.go:654] GET /healthz: (886.441µs) 0 [[monit/5.4] 127.0.0.1:37322]<br>
E0327 13:46:43.817523    4712 reflector.go:149] watch of *api.Node ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17177/7110]) [18176<br>
]<br>
I0327 13:46:46.492099    4712 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF<br>
I0327 13:46:46.492181    4712 iowatcher.go:91] Unexpected EOF during watch stream event decoding: unexpected EOF<br>
E0327 13:46:47.256804    4712 reflector.go:149] watch of *api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17354/6384]) [18<br>
353]<br>
E0327 13:46:47.256950    4712 reflector.go:149] watch of *api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [17352/8087]) [18351]<br>
I0327 13:46:50.558471    4712 kubelet.go:1099] No Infra Container for "my-hostname-density30-7bb294b4-d487-11e4-9336-a0481cabf39b8hwgv_default" found. All containers will be restarted.<br>
...<br>
[and it works since then].</p>
<p dir="auto">Investigate what is happening during that 45 seconds periods of "nothing".</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fgrzadkowski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fgrzadkowski">@fgrzadkowski</a></p>