<p dir="auto">While running density tests against a kubernetes cluster, we will occasionally run into nodes where the docker daemon becomes unresponsive.  Symptoms include:</p>
<ul dir="auto">
<li><code class="notranslate">docker ps</code> hangs</li>
<li><code class="notranslate">kernel: unregister_netdevice: waiting for veth0123456 to become free. Usage count = 1</code> appears in journal output</li>
</ul>
<p dir="auto">Probably an upstream issue.  But I would expect kubelet to notice that its docker deamon is unhealthy and send a 'NotReady' status back up to the controller manager.  Instead, it stays at 'Ready' and the scheduler assigns pods to it, which will stay stuck in Pending.</p>
<p dir="auto">This is not quite the same as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="114323676" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/16601" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/16601/hovercard" href="https://github.com/kubernetes/kubernetes/issues/16601">#16601</a> but our mitigation is the same, we're forced to reboot the node.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):<br>
test getsockopt: connection refused,<br>
getsockopt: connection refused,<br>
integration getsockopt: connection refused,<br>
integration test open file limit</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
NAME="Ubuntu"<br>
VERSION="16.04.1 LTS (Xenial Xerus)"<br>
ID=ubuntu<br>
ID_LIKE=debian<br>
PRETTY_NAME="Ubuntu 16.04.1 LTS"<br>
VERSION_ID="16.04"<br>
HOME_URL="<a href="http://www.ubuntu.com/" rel="nofollow">http://www.ubuntu.com/</a>"<br>
SUPPORT_URL="<a href="http://help.ubuntu.com/" rel="nofollow">http://help.ubuntu.com/</a>"<br>
BUG_REPORT_URL="<a href="http://bugs.launchpad.net/ubuntu/" rel="nofollow">http://bugs.launchpad.net/ubuntu/</a>"<br>
UBUNTU_CODENAME=xenial</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.4.0-47-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35537532" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/68" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/68/hovercard" href="https://github.com/kubernetes/kubernetes/pull/68">#68</a>-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Basically I am trying to run integration test on my local machine - as described here <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/devel/testing.md#integration-tests">https://github.com/kubernetes/kubernetes/blob/master/docs/devel/testing.md#integration-tests</a></p>
<p dir="auto">Lots of them failing with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="unable to sync kubernetes service: Post http://127.0.0.1:33526/api/v1/namespaces/default/services: dial tcp 127.0.0.1:33526: getsockopt: connection refused"><pre class="notranslate"><code class="notranslate">unable to sync kubernetes service: Post http://127.0.0.1:33526/api/v1/namespaces/default/services: dial tcp 127.0.0.1:33526: getsockopt: connection refused
</code></pre></div>
<p dir="auto">I have a fresh copy of the master branch - no local changes. When I attached to etcd via strace the following lines caught my attention:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="futex(0x1744568, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
...
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=64*1024}) = 0"><pre class="notranslate"><code class="notranslate">futex(0x1744568, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
...
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=64*1024}) = 0
</code></pre></div>
<p dir="auto">Seems like I’ve reached the limit of opened files on my machine which was set to 1024. Moreover when I increased the limit to 65535 <strong>ALL</strong> test went “green”.</p>
<p dir="auto">For a newcomer like me it was a challenging task just to get a fresh copy, compile it and run the test.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<ul dir="auto">
<li>
<p dir="auto">We could improve documentation - <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/devel/testing.md#integration-tests">https://github.com/kubernetes/kubernetes/blob/master/docs/devel/testing.md#integration-tests</a> and explicitly state that one have to set resources limit to X</p>
</li>
<li>
<p dir="auto">We could improve tests to set resources limit on behalf of a user - this requires root privileges</p>
</li>
<li>
<p dir="auto">I haven’t checked the test code but perhaps we could make the test code more polite and limit the number of the request we are sending to the server/etcd</p>
</li>
</ul>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Clone the repository, compile, run integration test</p>