<p dir="auto">This is a roundup issue for Kubernetes 1.4 and a tracker to orient folks on the ongoing work.  It is focused on all components that touch the API, but not necessarily the total system.  It also serves to identify areas where folks can get involved.</p>
<p dir="auto">Track issues using <a href="https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&amp;q=label%3Aarea%2Fperformance+label%3A%22team%2FCSI-API+Machinery+SIG%22+">labels performance and CSI-API-Machinery</a></p>
<h3 dir="auto">Ongoing efforts</h3>
<p dir="auto">Major focus: We want to get to our target cluster size metrics (5k nodes and 300k pods) in 1.4 or 1.5, and we are within an order of magnitude of that with 1.3 in many dimensions.</p>
<p dir="auto">Areas of attack: We are mostly CPU bound on the controllers and API servers today, and most of that is overhead due to garbage collection.  We also have had a fair number of contention areas that were algorithmic or lock related in the core components.  We've got a good feedback loop for identifying critical bottlenecks, but we could probably stand to get more profiles and metrics gathered from the high scale systems.</p>
<ul dir="auto">
<li>Reduce the CPU and garbage created by clients and the API server when handling API requests
<ul dir="auto">
<li>Introduce protobuf
<ul dir="auto">
<li>[complete <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="75730377" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/8132" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/8132/hovercard" href="https://github.com/kubernetes/kubernetes/issues/8132">#8132</a>] Used for most API objects when talking between client and server</li>
<li>[inprogress for 1.5] Use protobuf for storage format</li>
<li>Handle mixed content type storage (objects that embed other objects like Templates or third party resources)</li>
<li>[inprogress <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159852334" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27258" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/27258/hovercard" href="https://github.com/kubernetes/kubernetes/pull/27258">#27258</a>] Handle optional slices and maps correctly in protobuf</li>
<li><a href="https://github.com/gogo/protobuf/issues/174" data-hovercard-type="issue" data-hovercard-url="/gogo/protobuf/issues/174/hovercard">inprogress</a> Investigate use of string arena unmarshalling in protobuf to avoid []byte -&gt; string allocations in decode</li>
</ul>
</li>
<li>Optimize the conversion, defaulting, and decoding code paths
<ul dir="auto">
<li><a href="https://github.com/kubernetes/kubernetes/issues/25019" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/25019/hovercard">complete</a> Implement more efficient Quantity, using int64 and better serialization</li>
<li>[complete <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="151791703" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/24966" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/24966/hovercard" href="https://github.com/kubernetes/kubernetes/pull/24966">#24966</a>] Remove the use of reflection in conversion and defaulting
<ul dir="auto">
<li>Auto generate fast path conversion functions (AddGenericConversionFunc) for each API type</li>
</ul>
</li>
<li>[complete <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="150740424" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/24727" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/24727/hovercard" href="https://github.com/kubernetes/kubernetes/pull/24727">#24727</a>] Make conversions of identical types (builtins, maps, slices, pointers) an assignment, not a copy</li>
<li>[inprogress <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="161006493" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27659" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/27659/hovercard" href="https://github.com/kubernetes/kubernetes/pull/27659">#27659</a>] Further optimization of identical type conversion using unsafe.Pointer</li>
<li>[inprogress <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="178779790" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33352" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/33352/hovercard" href="https://github.com/kubernetes/kubernetes/pull/33352">#33352</a>] Split defaulting from conversion into its own function, possibly by generating an entire "Default_Pod" function from component parts, which allows us to avoid having to walk deep objects</li>
</ul>
</li>
<li>Use the versioned client where possible to avoid conversion altogether. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="168964772" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29934" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/29934/hovercard" href="https://github.com/kubernetes/kubernetes/issues/29934">#29934</a>
<ul dir="auto">
<li><a href="???">inprogress</a> Versioned client in controllers depends on the shared cache work (all controllers sharing a cache have to use the same object type). It's also blocked by the <a href="https://github.com/kubernetes/kubernetes/pull/29147" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/29147/hovercard">cutting a client library work</a>.</li>
</ul>
</li>
<li>Ensure that the Update and Create code paths are efficient (guaranteed update loop generates a lot of objects)</li>
<li>SelfLink and URL.String() are fairly expensive in the encode/decode path</li>
<li>For any JSON gaps (non-proto clients) we could look at easyjson, which appears to do more than ugorgji to reduce memory allocation during decode.</li>
</ul>
</li>
<li>Reduce the amount of duplicate objects being used by controller and admission caches and the amount of total memory used by caches
<ul dir="auto">
<li>[complete] Separate re-listing and re-syncing (former talks to api server, latter traverses cache) to reduce the amount of duplicate calls to the server and wasted conversion</li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/14978" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/14978/hovercard">complete</a> Add a SharedInformer and use it for Pod caches (all 8 of them)</li>
<li>[inprogress] Use shared Namespace caches in admission controllers</li>
<li>Investigate and identify other opportunities for shared caches</li>
<li>[inprogress] Investigate size of objects kept in shared and unshared caches for duplication and waste
<ul dir="auto">
<li>Pods tend to have larger objects <a href="https://github.com/coreos/kscale/pull/37/files#diff-a423e914cbd27c95d2c0b950e2e4ee5bR14">https://github.com/coreos/kscale/pull/37/files#diff-a423e914cbd27c95d2c0b950e2e4ee5bR14</a> than necessary - maps and quantity too large</li>
<li>Investigate using slices instead of maps for our many small maps, or having an <code class="notranslate">ArrayMap</code> equivalent to Java <a href="https://github.com/kubernetes/kubernetes/issues/31440" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/31440/hovercard">#31440</a></li>
<li>Investigate flyweight cache patterns for certain controllers</li>
</ul>
</li>
</ul>
</li>
<li>Identify unnecessary blocking / stalls in clients, watches, caches, and controllers
<ul dir="auto">
<li>[complete] Several fixes to 1.3 scheduler and caches</li>
<li>[complete] Avoid flushing aggressively in watches to reduces syscalls</li>
<li><a href="@deads2k">inprogress</a> Shared caches currently have coarse locks, investigate striping and reducing the duration of the locks</li>
<li>Investigate whether we have any slow cache reader problems with our shared caches</li>
<li>Audit the watch cache for locking issues</li>
<li>Better tooling and documentation to allow people to perform this sort of debugging</li>
<li>???</li>
</ul>
</li>
<li>Profile and optimize specific components and workloads
<ul dir="auto">
<li>Collect profile traces from kubemark and high scale clusters automatically</li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/18418" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/18418/hovercard">Scheduler</a> (lots of eyes on this, but also lots of algorithmic improvements possible)</li>
<li>Kubelet</li>
<li>[inprogress] Replication controller</li>
<li>Node controller (known issues with Timer, other garbage creation)</li>
<li>Quota
<ul dir="auto">
<li>[inprogress] Significant improvements including contention striping and batch assignment</li>
<li>Optimize and audit use of resource.Quantity in these codepaths</li>
</ul>
</li>
<li>Authentication / Authorization / RBAC (has been a significant source of perf bottlenecks in OpenShift in the more complex scenarios)</li>
<li>???</li>
</ul>
</li>
<li>Reduce traffic between components
<ul dir="auto">
<li>Investigation colocation of certain processes if necessary
<ul dir="auto">
<li>OpenShift offers all-in-one processes that bundle etcd, controller, apiserver, as does a few others, and there is some value in compact deployments having an efficient master (downsides are increased memory use and contention)</li>
<li><a href="https://github.com/openshift/origin/pull/8082" data-hovercard-type="pull_request" data-hovercard-url="/openshift/origin/pull/8082/hovercard">inprogress</a> Investigate loopback connections when in the same process (at the net.Chan level)</li>
<li>[mentioned by justinsb] Directly embed etcd and make calls to the Go API</li>
</ul>
</li>
<li>Reduce network bandwidth by sending less objects (not yet a blocker in any particular area)</li>
</ul>
</li>
</ul>
<p dir="auto">There is a lot of copied bash between the various providers.  We should create a shared library that multiple providers can use that'll simplify the amount of code for each provider.</p>