<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): cAdvisor inotify_init: too many open files</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: hardware</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): SUSE 12</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>: manul</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>: monit monitor kubelet and flanneld service, and kubelet service depends on flanneld service, i make flanneld not running, so kubelet will restart by monit, after many times restart, i fix flanneld service, but kubelet launch with error</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
kubelet running success</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E1123 14:47:21.372435   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for &quot;runtime&quot;: unable to find data for container /
E1123 14:47:21.372469   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: unable to find data for container /
I1123 14:47:25.806786   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:30.806954   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:31.397830   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for &quot;runtime&quot;: unable to find data for container /
E1123 14:47:31.397861   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: unable to find data for container /
I1123 14:47:35.807141   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:40.807326   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:41.415673   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for &quot;runtime&quot;: unable to find data for container /
E1123 14:47:41.415704   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: unable to find data for container /
I1123 14:47:45.807487   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]"><pre class="notranslate"><code class="notranslate">E1123 14:47:21.372435   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:21.372469   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:25.806786   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:30.806954   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:31.397830   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:31.397861   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:35.807141   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
I1123 14:47:40.807326   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
E1123 14:47:41.415673   24479 kubelet.go:2675] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1123 14:47:41.415704   24479 kubelet.go:2683] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
I1123 14:47:45.807487   24479 kubelet.go:2803] skipping pod synchronization - [Failed to start cAdvisor inotify_init: too many open files]
</code></pre></div>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):<br>
No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):<br>
docker hang<br>
docker ps hangs</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GKE v1.4.5</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
BUILD_ID=8872.26.0<br>
NAME="Container-VM Image"<br>
GOOGLE_CRASH_ID=Lakitu<br>
VERSION_ID=55<br>
VERSION=55<br>
ID=gci</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux gke-api-cluster-1-pool-v1-0f4fa4ec-p95f 4.4.21+ <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Wed Oct 26 13:00:52 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>: <code class="notranslate">docker ps</code> hangs for more than 10 seconds in <code class="notranslate">kube-docker-monitor.service</code> and the docker daemon is killed.</p>
<p dir="auto"><strong>What you expected to happen</strong>: <code class="notranslate">docker ps</code> to complete quickly and docker not to be killed.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible): I have not had success reproducing this at will.  It's a daily occurrence though.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
It feels weird to file this here as it's rather GKE-specific, but we had GCP support direct us here.</p>
<p dir="auto">I've collected several stack dumps here (taken after the 10s hang):<br>
<a href="https://gist.github.com/gdvalle/b58b5d793ebd229f9c40438a4c5ae160">https://gist.github.com/gdvalle/b58b5d793ebd229f9c40438a4c5ae160</a><br>
At least a couple of these seem to match up with this bug:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="184357224" data-permission-text="Title is private" data-url="https://github.com/moby/moby/issues/27607" data-hovercard-type="issue" data-hovercard-url="/moby/moby/issues/27607/hovercard" href="https://github.com/moby/moby/issues/27607">moby/moby#27607</a></p>
<p dir="auto">Looks like those patches should ship with the next 1.12 release...but no date for that yet.  However, 1.13 did just ship, and that looks like it has the patches.  So maybe we wait for k8s validation on that.</p>
<p dir="auto">So far it looks like we have a workaround by just allowing more time for the <code class="notranslate">docker ps</code> to succeed.  Existing containers on the node were still healthy so it seemed reasonable to bump it.<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/kubernetes/kubernetes/blob/8fd414537b5143ab039cb910590237cabf4af783/cluster/gce/gci/health-monitor.sh#L29">kubernetes/cluster/gce/gci/health-monitor.sh</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 29
      in
      <a data-pjax="true" class="commit-tease-sha" href="/kubernetes/kubernetes/commit/8fd414537b5143ab039cb910590237cabf4af783">8fd4145</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L29" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="29"></td>
          <td id="LC29" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-k">!</span> timeout 10 docker ps <span class="pl-k">&gt;</span> /dev/null<span class="pl-k">;</span> <span class="pl-k">then</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
We've upped the timeout to 50s, but over the last couple days we haven't seen it need more than about 35s in total with most being just a few seconds longer than 10.<p></p>