<p dir="auto">One of the reasons for keeping dead containers around in kubelet today is to get access to logs from previous instances. Dead container instances can take up disk space via their root filesystem and result in disk pressure on the nodes. To alleviate disk pressure and improve the logging experience in kubernetes, kubelet can retrieve logs from dead containers and GC them right away once kubelet doesn't depend on metadata associated with old containers.<br>
Specifically,</p>
<ul dir="auto">
<li>Kubelet can retrieve logs from the runtime and store in a per-pod, per-container directory inside of <code class="notranslate">/var/log/</code> directory. For the docker runtime, this can be a simple move operation. For rkt, we will have to retrieve the logs remotely.</li>
<li>The directory structure can be <code class="notranslate">/var/log/&lt;podUID&gt;/&lt;ContainerName&gt;/&lt;InstanceNumber&gt;_stdout.log</code></li>
<li>Update <code class="notranslate">kubectl logs -p</code> to return logs from the files instead of relying on the runtime for previous logs.</li>
<li>These logs will be kept around on a best-effort basis and will be deleted whenever there is disk pressure.</li>
<li>Kubelet can prefer keeping the first and most recent instances of a container around and aggressively delete other log files.</li>
<li>All these logs will be accessible initially via the <code class="notranslate">/logs</code> REST endpoint. In the future, we can consider expanding <code class="notranslate">kubectl logs</code> interface to support an instance number or add support for the first attempt specifically.</li>
</ul>
<p dir="auto">Monitoring/logging agents running on a node needs to know which pod a given container belongs to. Currently there is a lot of hacks around this:</p>
<ul dir="auto">
<li>filepath for logging</li>
<li>labels on docker</li>
<li>reverse lookup from api server</li>
</ul>
<p dir="auto">Each of them has obvious drawbacks. We need a consistent solution for this problem.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dchen1107/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dchen1107">@dchen1107</a> @kubernetes/sig-instrumentation @kubernetes/sig-node</p>