<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>"crypto/tls: failed to parse certificate PEM data"</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): v1.2.4</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;2&quot;, GitVersion:&quot;v1.2.4+coreos.cni.1&quot;, GitCommit:&quot;89e3af04ef436f049e2e2d169e57ad1712cac375&quot;, GitTreeState:&quot;clean&quot;"><pre class="notranslate"><code class="notranslate">Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4+coreos.cni.1", GitCommit:"89e3af04ef436f049e2e2d169e57ad1712cac375", GitTreeState:"clean"
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS EC2 Instance</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1010.4.0 (MoreOS)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): <code class="notranslate">Linux ip-10-0-2-42.ec2.internal 4.5.0-coreos-r1 #2 SMP Tue May 24 17:45:36 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux</code></li>
<li><strong>Install tools</strong>: <a href="https://github.com/coreos/coreos-kubernetes/blob/master/multi-node/aws/README.md"><code class="notranslate">kube-aws</code></a></li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
The <code class="notranslate">kubelet</code> service failed (reason unknown)<br>
Attempting to restart <code class="notranslate">kubelet</code> failed, with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="crypto/tls: failed to parse certificate PEM data"><pre class="notranslate"><code class="notranslate">crypto/tls: failed to parse certificate PEM data
</code></pre></div>
<p dir="auto">See full logs and rambling thoughts below.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Expected kubelet not to fail, and for kubelet to restart, if it did fail.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
?</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<h3 dir="auto">Full logs and rambling thoughts:</h3>
<p dir="auto">After noticing that the node was <em>NotReady</em></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ journalctl -u kubelet
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:01 UTC. --
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:01:27.793814    1276 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a8740298
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: , stderr: du: cannot access '/var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a874029848cc31e0a97d686a9bd3/upper/tmp/tmp-5039jZ3BGus1nfDW': No such file or directory
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: - exit status 1
Oct 16 12:48:40 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:48:40.543457    1276 event.go:193] Server rejected event '&amp;api.Event{TypeMeta:unversioned.TypeMeta{Kind:&quot;&quot;, APIVersion:&quot;&quot;}, ObjectMeta:api.ObjectMeta{Name:&quot;kube-dns-
Oct 16 18:55:04 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: W1016 18:55:04.175757    1276 request.go:627] Throttling request took 148.06449ms, request: GET:https://10.0.4.50:443/api/v1/nodes/ip-10-0-2-42.ec2.internal
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Job kubelet.service/start failed with result 'dependency'."><pre class="notranslate"><code class="notranslate">$ journalctl -u kubelet
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:01 UTC. --
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:01:27.793814    1276 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a8740298
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: , stderr: du: cannot access '/var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a874029848cc31e0a97d686a9bd3/upper/tmp/tmp-5039jZ3BGus1nfDW': No such file or directory
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: - exit status 1
Oct 16 12:48:40 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:48:40.543457    1276 event.go:193] Server rejected event '&amp;api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"kube-dns-
Oct 16 18:55:04 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: W1016 18:55:04.175757    1276 request.go:627] Throttling request took 148.06449ms, request: GET:https://10.0.4.50:443/api/v1/nodes/ip-10-0-2-42.ec2.internal
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Job kubelet.service/start failed with result 'dependency'.
</code></pre></div>
<p dir="auto">I'm not sure what caused kubelet to shut down. I see the <code class="notranslate">failed to collect filesystem stats</code> error, <a href="https://github.com/kubernetes/kubernetes/issues/16651#issuecomment-154145346" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/16651/hovercard">but it looks like that may be a red herring</a>, possibly caused by a custom docker garbage collection script we are running (because the built in cleanup services are not working for us).</p>
<p dir="auto">I see the <code class="notranslate">Job kubelet.service/start failed with result 'dependency'</code>. So check the docker service</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ journalctl -u docker 
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:41 UTC. --
Oct 16 18:58:05 ip-10-0-2-42.ec2.internal systemd[1]: Stopping Docker Application Container Engine...
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
Oct 17 00:00:14 ip-10-0-2-42.ec2.internal systemd[1]: Started Docker Application Container Engine.
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.249045066Z&quot; level=info msg=&quot;Graph migration to content-addressability took 0.00 seconds&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.252948977Z&quot; level=info msg=&quot;Firewalld running: false&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.566883833Z&quot; level=info msg=&quot;Loading containers: start.&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: .......................................................time=&quot;2016-10-17T00:00:26.766104356Z&quot; level=error msg=&quot;Error unmounting container 9c20ac08b141a474b8ba76698b339729907d35a6a286
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.767774056Z&quot; level=error msg=&quot;Error unmounting container e3274fd20b73356d2b9d4cf94cb2e5028b0cfdf46151002fd71af5bd86e4b14e: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.771743362Z&quot; level=error msg=&quot;Error unmounting container ea6d0e45e10db34b7b934a9307f1791336b258f125dee256d0aa9611363c5261: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.773098949Z&quot; level=error msg=&quot;Error unmounting container 7ad2c477853f1429ebefd7687af2b60ac73a83b2590fd52685e139dc073e1f6e: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.777013970Z&quot; level=error msg=&quot;Error unmounting container f2448c82bc67eb7b932b3ace75a878272ef917a09e37e542e7a540821815357b: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.788810168Z&quot; level=error msg=&quot;Error unmounting container 2ecdff73aedd2c17b57f784fc812ed715240627e37ff9b2ca6b6eb0fa323e757: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.799454818Z&quot; level=error msg=&quot;Error unmounting container f0e337d6fd15884d6dd8c69d7b8941232a332054a58d2c66ebc99eac0190a477: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.801740726Z&quot; level=error msg=&quot;Error unmounting container 512a8344fee1bb62bcddfa3327a03ea886ccf46fdd9c46f1a0d872ff386004b9: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.804142833Z&quot; level=error msg=&quot;Error unmounting container 5818f58156e378425cb2aa48e9646d0d35b86e89338616366b8b21f2fbe58e25: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.805297087Z&quot; level=error msg=&quot;Error unmounting container 20a4cda200c33d6263a540caba028a373ecdf5ca2733e40e32627efd71f38d79: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.807651272Z&quot; level=error msg=&quot;Error unmounting container 5e11bedd3511b33a1b16ec42af039c0f7a4959c86092f543f2f83816b564ce3f: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.808670518Z&quot; level=error msg=&quot;Error unmounting container 63d415b23d7d8d4f399f32d8e7cfa27148af80aa6ba3c1164cfbb43ce94f6a7e: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.809797868Z&quot; level=error msg=&quot;Error unmounting container f2199232be2d0f6f93f39e311970222e2d2e18bd8be62c4561304f9dcc58af3d: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.812997843Z&quot; level=error msg=&quot;Error unmounting container e260b44e03890339a4de616259d42b6f4529fa0a19e8e259b781d4abf7e68d1e: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.816434872Z&quot; level=error msg=&quot;Error unmounting container c23aa4d3db9cba77ee37d5debed98e66a09039ecb78c191c4372c187d9e69c3c: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.817642230Z&quot; level=error msg=&quot;Error unmounting container 0c0a0d62ca3af74660f6d671510823485cc1a37bb19e8718e93b28e4cf0c7e4f: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.818783758Z&quot; level=error msg=&quot;Error unmounting container 3015b62493fa80462a97336503fb7749cb5a28b37cf70073812b3d37fb806128: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.825780463Z&quot; level=error msg=&quot;Error unmounting container 8b549e1ffcd94240d9d7469bc20c400160b2b7ff9a8a3c44eccf48de8f62c141: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.868323368Z&quot; level=error msg=&quot;Error unmounting container caf802a7a506053a1f5eef98ca37a017f0203dffcc0eae68019c5288e165ac61: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.873743087Z&quot; level=error msg=&quot;Error unmounting container 66dbce340f4e2e0541c814ce224ed059fa99f13cd274a94b6e4ce6ab3a148e1d: not mounted&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.884863245Z&quot; level=info msg=&quot;Loading containers: done.&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.884879420Z&quot; level=info msg=&quot;Daemon has completed initialization&quot;
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.884896942Z&quot; level=info msg=&quot;Docker daemon&quot; commit=8acee1b execdriver=native-0.2 graphdriver=overlay version=1.10.3
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time=&quot;2016-10-17T00:00:26.890880322Z&quot; level=info msg=&quot;API listen on /var/run/docker.sock&quot;"><pre class="notranslate"><code class="notranslate">$ journalctl -u docker 
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:41 UTC. --
Oct 16 18:58:05 ip-10-0-2-42.ec2.internal systemd[1]: Stopping Docker Application Container Engine...
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
Oct 17 00:00:14 ip-10-0-2-42.ec2.internal systemd[1]: Started Docker Application Container Engine.
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.249045066Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.252948977Z" level=info msg="Firewalld running: false"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.566883833Z" level=info msg="Loading containers: start."
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: .......................................................time="2016-10-17T00:00:26.766104356Z" level=error msg="Error unmounting container 9c20ac08b141a474b8ba76698b339729907d35a6a286
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.767774056Z" level=error msg="Error unmounting container e3274fd20b73356d2b9d4cf94cb2e5028b0cfdf46151002fd71af5bd86e4b14e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.771743362Z" level=error msg="Error unmounting container ea6d0e45e10db34b7b934a9307f1791336b258f125dee256d0aa9611363c5261: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.773098949Z" level=error msg="Error unmounting container 7ad2c477853f1429ebefd7687af2b60ac73a83b2590fd52685e139dc073e1f6e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.777013970Z" level=error msg="Error unmounting container f2448c82bc67eb7b932b3ace75a878272ef917a09e37e542e7a540821815357b: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.788810168Z" level=error msg="Error unmounting container 2ecdff73aedd2c17b57f784fc812ed715240627e37ff9b2ca6b6eb0fa323e757: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.799454818Z" level=error msg="Error unmounting container f0e337d6fd15884d6dd8c69d7b8941232a332054a58d2c66ebc99eac0190a477: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.801740726Z" level=error msg="Error unmounting container 512a8344fee1bb62bcddfa3327a03ea886ccf46fdd9c46f1a0d872ff386004b9: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.804142833Z" level=error msg="Error unmounting container 5818f58156e378425cb2aa48e9646d0d35b86e89338616366b8b21f2fbe58e25: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.805297087Z" level=error msg="Error unmounting container 20a4cda200c33d6263a540caba028a373ecdf5ca2733e40e32627efd71f38d79: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.807651272Z" level=error msg="Error unmounting container 5e11bedd3511b33a1b16ec42af039c0f7a4959c86092f543f2f83816b564ce3f: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.808670518Z" level=error msg="Error unmounting container 63d415b23d7d8d4f399f32d8e7cfa27148af80aa6ba3c1164cfbb43ce94f6a7e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.809797868Z" level=error msg="Error unmounting container f2199232be2d0f6f93f39e311970222e2d2e18bd8be62c4561304f9dcc58af3d: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.812997843Z" level=error msg="Error unmounting container e260b44e03890339a4de616259d42b6f4529fa0a19e8e259b781d4abf7e68d1e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.816434872Z" level=error msg="Error unmounting container c23aa4d3db9cba77ee37d5debed98e66a09039ecb78c191c4372c187d9e69c3c: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.817642230Z" level=error msg="Error unmounting container 0c0a0d62ca3af74660f6d671510823485cc1a37bb19e8718e93b28e4cf0c7e4f: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.818783758Z" level=error msg="Error unmounting container 3015b62493fa80462a97336503fb7749cb5a28b37cf70073812b3d37fb806128: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.825780463Z" level=error msg="Error unmounting container 8b549e1ffcd94240d9d7469bc20c400160b2b7ff9a8a3c44eccf48de8f62c141: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.868323368Z" level=error msg="Error unmounting container caf802a7a506053a1f5eef98ca37a017f0203dffcc0eae68019c5288e165ac61: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.873743087Z" level=error msg="Error unmounting container 66dbce340f4e2e0541c814ce224ed059fa99f13cd274a94b6e4ce6ab3a148e1d: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884863245Z" level=info msg="Loading containers: done."
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884879420Z" level=info msg="Daemon has completed initialization"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884896942Z" level=info msg="Docker daemon" commit=8acee1b execdriver=native-0.2 graphdriver=overlay version=1.10.3
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.890880322Z" level=info msg="API listen on /var/run/docker.sock"
</code></pre></div>
<p dir="auto">I see a <code class="notranslate">docker.service: Job docker.service/start failed with result 'dependency'.</code> log in there. Maybe it's flannel?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ journalctl -u flanneld           
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:53:22 UTC. --
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080742 00001 device.go:187] calling NeighSet: 10.2.58.14, 96:75:16:ae:1e:ce
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080863 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.407509 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.832713 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.408852 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.856828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.032862 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.410953 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.816900 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.896856 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:52 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:52.752864 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.035102 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.038654 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.633088 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.902816 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.040702 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.080715 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.720732 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.896893 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.904658 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.042682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427159 00001 vxlan.go:345] L3 miss: 10.2.59.9
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427208 00001 device.go:187] calling NeighSet: 10.2.59.9, be:66:1c:f3:6f:bf
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427391 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.758828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.906672 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395214 00001 vxlan.go:345] L3 miss: 10.2.58.3
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395269 00001 device.go:187] calling NeighSet: 10.2.58.3, 96:75:16:ae:1e:ce
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395376 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.560665 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.760705 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996054 00001 vxlan.go:345] L3 miss: 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996108 00001 device.go:187] calling NeighSet: 10.2.59.8, be:66:1c:f3:6f:bf
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996212 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.086698 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.726688 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.762682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.902809 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.088875 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.727838 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.729197 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.752964 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.904853 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.090678 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.566701 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.728951 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.730772 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.906840 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112784 00001 vxlan.go:345] L3 miss: 10.2.58.10
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112832 00001 device.go:187] calling NeighSet: 10.2.58.10, 96:75:16:ae:1e:ce
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112943 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.568835 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.731082 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.570500 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.758877 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:17:02 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:02.027092 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0"><pre class="notranslate"><code class="notranslate">$ journalctl -u flanneld           
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:53:22 UTC. --
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080742 00001 device.go:187] calling NeighSet: 10.2.58.14, 96:75:16:ae:1e:ce
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080863 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.407509 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.832713 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.408852 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.856828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.032862 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.410953 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.816900 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.896856 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:52 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:52.752864 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.035102 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.038654 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.633088 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.902816 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.040702 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.080715 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.720732 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.896893 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.904658 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.042682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427159 00001 vxlan.go:345] L3 miss: 10.2.59.9
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427208 00001 device.go:187] calling NeighSet: 10.2.59.9, be:66:1c:f3:6f:bf
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427391 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.758828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.906672 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395214 00001 vxlan.go:345] L3 miss: 10.2.58.3
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395269 00001 device.go:187] calling NeighSet: 10.2.58.3, 96:75:16:ae:1e:ce
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395376 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.560665 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.760705 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996054 00001 vxlan.go:345] L3 miss: 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996108 00001 device.go:187] calling NeighSet: 10.2.59.8, be:66:1c:f3:6f:bf
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996212 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.086698 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.726688 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.762682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.902809 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.088875 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.727838 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.729197 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.752964 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.904853 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.090678 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.566701 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.728951 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.730772 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.906840 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112784 00001 vxlan.go:345] L3 miss: 10.2.58.10
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112832 00001 device.go:187] calling NeighSet: 10.2.58.10, 96:75:16:ae:1e:ce
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112943 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.568835 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.731082 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.570500 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.758877 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:17:02 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:02.027092 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
</code></pre></div>
<p dir="auto">Don't see any fatal errors there...</p>
<p dir="auto">So I try restarting <code class="notranslate">kubelet</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ servicectl restart kubelet
$ journalctl -fu kubelet
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887527    7992 manager.go:123] Starting to sync pod status with apiserver
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887555    7992 kubelet.go:2356] Starting kubelet main sync loop.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887568    7992 kubelet.go:2365] skipping pod synchronization - [container runtime is down]
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: F1017 14:02:37.888926    7992 server.go:118] crypto/tls: failed to parse certificate PEM data
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Unit entered failed state.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Failed with result 'exit-code'."><pre class="notranslate"><code class="notranslate">$ servicectl restart kubelet
$ journalctl -fu kubelet
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887527    7992 manager.go:123] Starting to sync pod status with apiserver
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887555    7992 kubelet.go:2356] Starting kubelet main sync loop.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887568    7992 kubelet.go:2365] skipping pod synchronization - [container runtime is down]
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: F1017 14:02:37.888926    7992 server.go:118] crypto/tls: failed to parse certificate PEM data
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Unit entered failed state.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Failed with result 'exit-code'.
</code></pre></div>
<p dir="auto">That <code class="notranslate">crypto/tls: failed to parse certificate PEM data</code> message looks ominous.</p>
<p dir="auto">I heard from a colleague that on a different node, the RSA keys just disappeared. He was able to fix the issue by copying them in from another server. I don't know if that's what's going on here, but just a guess... (and he's out of town, so I can't verify)</p>
<p dir="auto">All of our pods were quickly relocated to a functioning node, so the impact wasn't too heavy. But it would be nice to know what's going on, and how I could fix/prevent this in the future.</p>
<p dir="auto">As discussed in <a href="https://groups.google.com/forum/#!topic/google-containers/vA8vIfZ7WUA" rel="nofollow">https://groups.google.com/forum/#!topic/google-containers/vA8vIfZ7WUA</a>, we can make some small changes to allow one yaml file contain multiple resources to reduce the file amount of a complex system.</p>