<p dir="auto">FEATURE REQUEST: Per destination bandwidth shaping</p>
<p dir="auto">While it's now possible to add a per-pod bandwidth limit with annotations <code class="notranslate">kubernetes.io/{ingress,egress}-bandwidth</code> this limit is set on the pod regardless of the trafic's origin/destination.</p>
<p dir="auto">In our use-case we would like to host containers for other companies in our Kubernetes cluster. While we would like the local&lt;-&gt;local bandwidth to be unlimited, we want to limit the internet-egress bandwidth of theses pods to avoid paying egress charges on them.<br>
Another option would be per-deployment internet-egress accounting but I couldn't find anything on this.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):<br>
bandwidth:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="97852713" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/11965" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/11965/hovercard" href="https://github.com/kubernetes/kubernetes/issues/11965">#11965</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51651113" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/2856" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/2856/hovercard" href="https://github.com/kubernetes/kubernetes/issues/2856">#2856</a></p>
<p dir="auto">shaper:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="98002273" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/11988" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/11988/hovercard" href="https://github.com/kubernetes/kubernetes/pull/11988">#11988</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="145232099" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/23740" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/23740/hovercard" href="https://github.com/kubernetes/kubernetes/pull/23740">#23740</a></p>
<hr>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.3.5</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Google GKE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Debian Wheezy</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 3.16.0-4-amd64</li>
</ul>
<p dir="auto">Hi. I'm writing this partially because I want to start a discussion and to leave some trails for others potentially affected.</p>
<p dir="auto">I started my journey with Kubernetes just a few days ago. I downloaded the 1.0.6 release (now I know this a only a pre-release and I should've downloaded the 1.0.4 release but this doesn't seem to be related to my particular issue).</p>
<p dir="auto">I brought up a tiny k8s cluster with the following config:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="export KUBERNETES_PROVIDER=aws
export KUBE_AWS_ZONE=eu-west-1a
export NUM_MINIONS=3
export MINION_SIZE=t2.micro
export AWS_S3_REGION=eu-west-1
export AWS_S3_BUCKET=xxx
export KUBE_AWS_INSTANCE_PREFIX=xxx"><pre class="notranslate"><code class="notranslate">export KUBERNETES_PROVIDER=aws
export KUBE_AWS_ZONE=eu-west-1a
export NUM_MINIONS=3
export MINION_SIZE=t2.micro
export AWS_S3_REGION=eu-west-1
export AWS_S3_BUCKET=xxx
export KUBE_AWS_INSTANCE_PREFIX=xxx
</code></pre></div>
<p dir="auto">My cluster was created properly and after a bit of playing I left it in a good shape.</p>
<p dir="auto">The next day I wanted to investigate some DNS related issue on k8s nodes so I logged in on one of the minions. I noticed <code class="notranslate">*** System restart required ***</code> in the MOTD. I ignored it for a moment and investigated the DNS matters (some <code class="notranslate">nslookup</code>s, nothing dangerous). Then I rebooted the machine.</p>
<p dir="auto">Once the machine was up again I found out that <code class="notranslate">kubectl</code> reported it as <code class="notranslate">NotReady</code>. I started digging and investigating. It took me some time to find out that the Docker service on the node was not running:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sudo docker info
FATA[0001] Get http:///var/run/docker.sock/v1.18/info: read unix /var/run/docker.sock: connection reset by peer. Are you trying to connect to a TLS-enabled daemon without TLS?"><pre class="notranslate"><code class="notranslate">$ sudo docker info
FATA[0001] Get http:///var/run/docker.sock/v1.18/info: read unix /var/run/docker.sock: connection reset by peer. Are you trying to connect to a TLS-enabled daemon without TLS?
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sudo journalctl -u docker -f
-- Logs begin at Fri 2015-09-18 08:06:53 UTC. --
Sep 18 08:09:49 ip-172-20-0-149 systemd[1]: docker.service failed.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service holdoff time over, scheduling restart.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Started Docker Application Container Engine.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Starting Docker Application Container Engine...
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time=&quot;2015-09-18T08:09:51Z&quot; level=info msg=&quot;+job serveapi(fd://)&quot;
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time=&quot;2015-09-18T08:09:51Z&quot; level=info msg=&quot;Listening for HTTP on fd ()&quot;
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time=&quot;2015-09-18T08:09:51Z&quot; level=fatal msg=&quot;Shutting down daemon due to errors: error intializing graphdriver: driver not supported&quot;
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Unit docker.service entered failed state.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service failed."><pre class="notranslate"><code class="notranslate">$ sudo journalctl -u docker -f
-- Logs begin at Fri 2015-09-18 08:06:53 UTC. --
Sep 18 08:09:49 ip-172-20-0-149 systemd[1]: docker.service failed.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service holdoff time over, scheduling restart.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Started Docker Application Container Engine.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Starting Docker Application Container Engine...
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time="2015-09-18T08:09:51Z" level=info msg="+job serveapi(fd://)"
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time="2015-09-18T08:09:51Z" level=info msg="Listening for HTTP on fd ()"
Sep 18 08:09:51 ip-172-20-0-149 docker[3397]: time="2015-09-18T08:09:51Z" level=fatal msg="Shutting down daemon due to errors: error intializing graphdriver: driver not supported"
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: Unit docker.service entered failed state.
Sep 18 08:09:51 ip-172-20-0-149 systemd[1]: docker.service failed.
</code></pre></div>
<p dir="auto">Further investigation led me to a conclusion that the problem arose from the fact that before the reboot the kernel was upgraded to 3.19.0-28, which no longer by default supports AUFS module which Docker uses as its storage driver.</p>
<p dir="auto">Looks like the solution is to install the <code class="notranslate">linux-image-extra-virtual</code>, as concluded from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="57964797" data-permission-text="Title is private" data-url="https://github.com/moby/moby/issues/10859" data-hovercard-type="issue" data-hovercard-url="/moby/moby/issues/10859/hovercard" href="https://github.com/moby/moby/issues/10859">moby/moby#10859</a> and other sources, and reboot the node. After reboot the node got back to the <code class="notranslate">Ready</code> state.</p>
<p dir="auto">Moreover, I guess an upgrade to a newer Docker version, using the official install command <code class="notranslate">curl -sSL https://get.docker.com/ | sh</code> would solve the problem as the most recent install script already installs <code class="notranslate">linux-image-extra-virtual</code> as per <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="57966342" data-permission-text="Title is private" data-url="https://github.com/moby/moby/issues/10860" data-hovercard-type="pull_request" data-hovercard-url="/moby/moby/pull/10860/hovercard" href="https://github.com/moby/moby/pull/10860">moby/moby#10860</a>, although I'm not sure how Docker was installed on the k8s node, so I don't know if it's safe to upgrade it this way.</p>
<p dir="auto">So I have a number of questions:</p>
<ol dir="auto">
<li>should k8s provide a workaround / fix for this particular issue?</li>
<li>or is it left to a cluster admin as it's an issue of a particular provider/env/configuration?</li>
<li>why is k8s using Docker v1.6.0 which is pretty old?</li>
<li>is it safe to use k8s in production on a setup in which Docker uses AUFS storage driver which seems to be deprecated by the Ubuntu team and thus dropped from the main kernel image?</li>
</ol>
<p dir="auto">Note that I haven't done anything unusual, just installed fresh k8s on AWS using the default Ubuntu release. Hence I believe it's important to fix it somehow in k8s.</p>
<p dir="auto">Possible duplicate: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="88226425" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/9779" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/9779/hovercard" href="https://github.com/kubernetes/kubernetes/issues/9779">#9779</a>.</p>