<p dir="auto">Per <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/quinton-hoole/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/quinton-hoole">@quinton-hoole</a>'s request, I'm attempting to chart a well established path such that we can address some of the larger communications issues.  Hopefully this can serve as a jump point for the communications issues we wish to target in the 1.3 cycle.</p>
<h2 dir="auto">Problem Statement</h2>
<p dir="auto">The <strong>overhead</strong> and data flow of the kubernetes control plane has led to a series of artificial limits, or <strong>caps</strong> (qps,max-inflight, etc), that have been placed on different system components.  These caps are an attempt to limit the system from overloading the api-server, and to ensure certain SLOs are met.  The core issue is not that the caps exist, but the fact that the interaction between the components can cause system overload in the first place.</p>
<p dir="auto">This epic, targeted for 1.3, is intended to help breakdown some of the different facets of this issue as we see them today, and address each aspect independently, provided there are resources available to do so.</p>
<p dir="auto">The higher level objective would be to fix the system where possible, in order to prevent the degenerative case of overloading from occurring... at all.</p>
<h2 dir="auto">Breakdown</h2>
<p dir="auto">From operations and tests, we have witnessed the following issues which relate to this broader epic:</p>
<p dir="auto"><strong>P1:</strong> There are a large number of connections in the system. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="133041411" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/21081" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/21081/hovercard" href="https://github.com/kubernetes/kubernetes/issues/21081">#21081</a>)</p>
<ul dir="auto">
<li><em>Worst offenders are not quantified?</em></li>
<li><strong>A1.1:</strong> Analyze api-server metrics, and add metrics on who is creating the most connections, investigate their usage, and optimize their workflow.  Different solutions may exist depending on the situation, which in turn may become their own issues.  (This is a meta-problem).</li>
<li><em>Each new connection incurs an overhead in the system (TLS+kernel+application), and there isn't a single choke-point to throttle during overloaded situations.</em></li>
<li><strong>A1.2:</strong> Pursue http2 for internal communications, each component should have only 1, or 2 connections to the main api-server, as well as golang 1.6 for the decreased EC/TLS overhead.</li>
</ul>
<p dir="auto"><strong>P2:</strong> Components do not properly back-off under extraneous load conditions.</p>
<ul dir="auto">
<li><em>We can create a number of conditions which can cause the system to overload, e.g. fail-over, or aggressive loading.</em></li>
<li><strong>A2.1:</strong> Once we've addressed P1, and have a single choke-point it's relatively trivial to back-off during loaded situations.</li>
</ul>
<p dir="auto"><strong>P3</strong>:  Enconding and decoding dominate api-server load</p>
<ul dir="auto">
<li><strong>A3.1:</strong> We've known this for a long time, and there are a series of issues around protobuf and gRPC to address. <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wojtek-t/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wojtek-t">@wojtek-t</a> has an issue list.</li>
</ul>
<p dir="auto">/cc <a class="team-mention js-team-mention notranslate" data-error-text="Failed to load team members" data-id="3959732" data-permission-text="Team members are private" data-url="/orgs/kubernetes/teams/sig-scalability/members" data-hovercard-type="team" data-hovercard-url="/orgs/kubernetes/teams/sig-scalability/hovercard" href="https://github.com/orgs/kubernetes/teams/sig-scalability">@kubernetes/sig-scalability</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jeremyeder/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jeremyeder">@jeremyeder</a></p>
<p dir="auto">Currently we manually convert the yaml/json files embedded in markdown files. To automate this process, we propose to add features to the kube-version-change tool:</p>
<ul dir="auto">
<li>we need to remove the default fields in the output of kube-version-change tool</li>
<li>we need to add a flag to kube-version-change that commands the tool to recursively convert all the .md files in a directory</li>
</ul>
<p dir="auto">Some code that maybe reused:</p>
<ul dir="auto">
<li><code class="notranslate">sampleRegexp</code> in examples/examples_test.go matches yaml/json block in markdown files</li>
</ul>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikhiljindal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikhiljindal">@nikhiljindal</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/krousey/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/krousey">@krousey</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bgrant0607/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bgrant0607">@bgrant0607</a> could you @ more folks? Thank you.</p>