<h2 dir="auto">Objective</h2>
<p dir="auto">The purpose of this document is to consolidated the major Kubernetes Volume issues/feature requests, and introduce high-level designs that ensure alignment between all of them.</p>
<h2 dir="auto">Background</h2>
<p dir="auto">There are three big designs currently being considered for Kubernetes Volume Storage:</p>
<ol dir="auto">
<li>Dynamic Provisioning of Volumes
<ul dir="auto">
<li>Design proposal by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/markturansky/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/markturansky">@markturansky</a> in issue <a href="https://github.com/kubernetes/kubernetes/pull/17056" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/17056/hovercard">#17056</a></li>
</ul>
</li>
<li>Volume Attach Detach Controller
<ul dir="auto">
<li>Design proposal by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/saad-ali/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/saad-ali">@saad-ali</a> in issue <a href="https://github.com/kubernetes/kubernetes/issues/15524" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/15524/hovercard">#15524</a></li>
</ul>
</li>
<li>Flex Volume Plugin
<ul dir="auto">
<li>Proposed implementation by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/NELCY/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/NELCY">@NELCY</a> in PR <a href="https://github.com/kubernetes/kubernetes/pull/13840" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/13840/hovercard">#13840</a></li>
</ul>
</li>
</ol>
<p dir="auto">The solutions to all three can be considered and designed in isolation, however they touch on overlapping issues. Therefore, it makes sense to first agree on an overall design direction for the Kubernetes Volume architecture, before finalizing on the the specifics of each of these.</p>
<h2 dir="auto">Goals</h2>
<ul dir="auto">
<li>Introduce Dynamic Provisioning
<ul dir="auto">
<li>Introduce ability for cluster to automatically (dynamically) create (provision) volumes instead of relying on a cluster admin to manually create them ahead of time.</li>
<li><em>Update (Jul 1, 2016):</em> Introduced in v1.2 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="108262809" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/14537" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/14537/hovercard" href="https://github.com/kubernetes/kubernetes/pull/14537">#14537</a>) as alpha. Being fully flushed out in v1.4 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158772696" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26908" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/26908/hovercard" href="https://github.com/kubernetes/kubernetes/pull/26908">#26908</a>).</li>
</ul>
</li>
<li>Make Attach/Detach Robust
<ul dir="auto">
<li>Make volume attachment and detachment independent of individual node availability.</li>
<li><em>Update (Jul 1, 2016):</em> Introduced in v1.3 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="156938340" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26351" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/26351/hovercard" href="https://github.com/kubernetes/kubernetes/pull/26351">#26351</a>).</li>
</ul>
</li>
<li>Improve Plugin Model
<ul dir="auto">
<li>Establish a plugin model that allows new storage plugins to be written without recompiling Kubernetes code.</li>
<li><em>Update (Jul 1, 2016):</em> discussion is <a href="https://groups.google.com/forum/#!topic/kubernetes-sig-storage/9o1vA4jFwqk" rel="nofollow">ongoing</a>.</li>
</ul>
</li>
<li>Enable Modularity and Improve Deployment
<ul dir="auto">
<li>Make it easy for cluster admins to select which volume types they want to support, and easily configure/deploy the plugin(s) across the cluster.</li>
</ul>
</li>
<li>Volume Selector
<ul dir="auto">
<li>Enable users to select specific storage based on some implementation specific characteristics (specific rack, etc.).</li>
<li><em>Update (Jul 1, 2016):</em> Introduced in v1.3 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="155843037" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/25917" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/25917/hovercard" href="https://github.com/kubernetes/kubernetes/pull/25917">#25917</a>.</li>
</ul>
</li>
<li>Volume Classes
<ul dir="auto">
<li>Create an abstraction layer (classes) for Kubernetes volumes users so they can request different grades of storage without worrying about the specifics of any one storage implementation.</li>
<li><em>Update (Jul 1, 2016):</em> Being fully flushed out as part of dynamic provisioning design in v1.4 (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158772696" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26908" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/26908/hovercard" href="https://github.com/kubernetes/kubernetes/pull/26908">#26908</a>).</li>
</ul>
</li>
</ul>
<h2 dir="auto">Proposal 1: Single Controller</h2>
<p dir="auto">Replace the existing persistent volume controllers that do binding (volumes to claims) and recycling (wiping volumes for reuse), with one controller that does <strong>provisioning</strong> (creating new volumes), binding, and recycling. This controller could monitor <code class="notranslate">PersistentVolume</code> and <code class="notranslate">PersistentVolumeClaim</code> objects via the API server to determine when a new volume needs to be created.</p>
<p dir="auto">Another new controller would be responsible for <strong>attaching and detaching</strong> volumes by monitoring <code class="notranslate">PersistentVolume</code> and <code class="notranslate">Pod</code> objects to determine when to attach/detach a volume from the node.</p>
<p dir="auto">For a seamless transition, in the immediate term, the existing volume <strong>plugin model</strong> could be used largely as is (although some refactoring would be needed). To implement a storage plugin a third party would need to provide methods to create/delete/attach/detach and optionally mount/unmount a volume. The new controllers and kubelet would call out to these methods as needed.</p>
<p dir="auto">The Flex Volume plugin (which adds support for exec based plugins), would be a first step towards enabling creators of third party storage plugin to add support for new volume types without having to add code to Kubernetes. The problem with the Flex plugin is that it doesn’t provide a good deployment mechanism. In order to support a new volume plugin, an admin needs to drop scripts into the correct directories on each node. However, once all the volume controllers move to the master, the exec scripts would only need to be dropped on to the master (not the nodes, since the flex plugin currently doesn’t support custom mount/unmount logic).</p>
<p dir="auto"><em>Longer term all third party plugin code would be containerized and removed from the Kubernetes code base.</em> The controllers on master could “docker run” the plugin container with the correct parameters to create/delete/attach/detach volumes. In order to support fully custom mount/unmount logic, the containerized plugin could contain mount/unmount code that Kubelet could similarly “docker run” (this will have to wait until Docker makes the changes needed to allow an executable running inside a container to mount to the host).</p>
<p dir="auto">The controllers and kubelet will need need to know which plugin container corresponds to which volume plugin and the other plugin specific configuration information. This will require the cluster to maintain a one to one mapping of volume-type strings to a plugin container path + any cluster wide plugin configuration for that plugin (e.g. <code class="notranslate">GCE</code> maps to <code class="notranslate">{container: “k8s-volume-plugin-gce”, maxVolumesToProvision: 20, … }</code>). This mapping must be available to both the controllers and kubelet (it could be stored in config data or a new API object).</p>
<p dir="auto">When a cluster administrator wants to add support for a new volume-type they would simply add a new key-value pair to this mapping.</p>
<h3 dir="auto">Pros:</h3>
<ul dir="auto">
<li>More concrete contract between Kubernetes and volume plugins.</li>
<li>Easier to implement volume plugins
<ul dir="auto">
<li>Plugin writers just have to provide implementations for methods like create, attach, detach, etc. and NOT controller logic which is more complicated.</li>
</ul>
</li>
<li>All plugins benefit from fixes/improvements to common controllers.</li>
<li>More performant to run a single controller vs n controllers.</li>
<li>Smaller incremental steps required to get to proposal’s end state.</li>
</ul>
<h3 dir="auto">Cons:</h3>
<ul dir="auto">
<li>Kubernetes core still responsible for maintaining a lot of volume logic.</li>
<li>Less flexibility for individual plugins (i.e will the API be sufficient for all possible volume plugins?).</li>
</ul>
<h2 dir="auto">Proposal 2: Multiple Controllers</h2>
<p dir="auto">Alternatively, instead of a set of Kubernetes provided controllers, the entire controller implementation could be left up to individual plugins. So each plugin would have a controller that would monitor the API Server for PV, PVC, and Pod objects, and when it finds a PVC that it can fulfill, it will claim it, bind it, and be responsible for it through attachment, detachment, all the way to recycling.</p>
<p dir="auto">Similar to the other proposal, plugins should be containerized. But instead of the container containing a binary that is triggered for specific tasks (attach, provision, etc.), it would contain the entire controller and run for the life of the cluster (or until the volume type is no longer needed) maybe via a replication controller (to ensure availability). Fully custom mount/unmount logic could be supported similar to the other proposal: by containerizing the mount/unmount code and having Kubelet execute it via a “docker run” (requires Docker support).</p>
<h3 dir="auto">Pros:</h3>
<ul dir="auto">
<li>More flexibility for plugin writers.
<ul dir="auto">
<li>Authors can decide how to implement pretty much everything.</li>
</ul>
</li>
<li>Less Kubernetes exposure.
<ul dir="auto">
<li>Since the plugin configuration can be passed directly to the controller rather than through kubernetes, that’s one less thing that must be added to config data or a new API object.</li>
</ul>
</li>
<li>Slightly simpler installation
<ul dir="auto">
<li>Support for a plugin can be added simply by running the plugin container. (Although the other proposal is not much more complicated.)</li>
</ul>
</li>
<li>Aligns with Kubernetes Networking plugin design</li>
</ul>
<h3 dir="auto">Cons:</h3>
<ul dir="auto">
<li>Would lead to code duplication.
<ul dir="auto">
<li>Most controllers will likely be implementing very similar logic for the controller portion, leading to code duplication/proliferation of bugs.</li>
<li>Could be worked around by creating a library that plugin writers could use for standard implementations.</li>
</ul>
</li>
<li>Less performant.
<ul dir="auto">
<li>Having to run a controller for each plugin adds overhead, especially for small clusters.</li>
</ul>
</li>
</ul>
<h2 dir="auto">Comparison</h2>
<h3 dir="auto">Dynamic Provisioning and Attach/Detach Controller</h3>
<p dir="auto">Differs for each proposal: Single controller vs multiple controller. See proposals above for details.</p>
<h3 dir="auto">Improve Plugin Model</h3>
<p dir="auto">Differs for each proposal. See proposals above for details.</p>
<h3 dir="auto">Enable Modularity and Improve Deployment</h3>
<p dir="auto">Once all plugins are containerized as both plans propose, we should get the kind of plugin modularity we’re hoping for.</p>
<p dir="auto">To deploy a new volume-type the cluster administrator would have to:</p>
<ul dir="auto">
<li>Installing plugin client binaries on each node.
<ul dir="auto">
<li>Short term: the admin must manually copy these files.</li>
<li>Long term: these should be containerized and deployed as deamonset pods.</li>
</ul>
</li>
<li>Configure volume classes (optional)--see below for details.
<ul dir="auto">
<li>Short term: Config data created and maintained by admin.</li>
<li>Long term: Kubectl command for adding and removing classes that does this automatically.</li>
</ul>
</li>
</ul>
<p dir="auto">In addition, only for proposal 1, the cluster admin would have to:</p>
<ul dir="auto">
<li>Add an entry to the key-value list keeping track of supported plugins to plugin container path/plugin config.
<ul dir="auto">
<li>Short term: Config data manually updated by admin.</li>
<li>Long term: Kubectl command for adding and removing supported plugins (i.e. admin specifies a volume-type and config, and Kubectl automatically adds it to Config Data).</li>
</ul>
</li>
</ul>
<p dir="auto">And, for proposal 2, the cluster admin would have to:</p>
<ul dir="auto">
<li>Start the controller container (maybe as a replication controller) with the correct parameters (plugin config) so that it can start watching for PVCs, etc.</li>
</ul>
<h3 dir="auto">Volume Selector</h3>
<p dir="auto">Volume selection support can be added the same way regardless of which proposal is implemented. Add “Labels” to the <code class="notranslate">PersistentVolume</code> object and “Selectors” to the <code class="notranslate">PersistentVolumeClaim</code> object. The binding controller(s) must resolve labels as it tries to fulfill PVCs.</p>
<h3 dir="auto">Volume Classes</h3>
<p dir="auto">Volume Classes are a way to create an abstraction layer over Kubernetes volumes so that users can request different grades of storage (classes) without worrying about the specifics of any one storage implementation.</p>
<p dir="auto">Implementation of classes can not be pushed into the plugin, because if individual plugins define the classes they support, then the whole point of the abstraction is lost. Instead, the cluster administrator must define the set of classes the cluster should support and the mapping of classes to the “knobs” exposed by individual plugins. More concretely, this means, for both proposals, that the cluster must maintain a mapping of admin-defined class strings to a list of parameters for each plugin that fulfills that class (e.g. “Gold” maps to “GCE plugin with parameter SSD” and “NFS with parameter XYZ”). This mapping must be maintained outside of the plugin (maybe in config data or a new API object).</p>
<p dir="auto">Weather the blob is a simple string, a list of key-value pairs, or structured JSON is up for debate. It can be argued that a simple string or key-value pair list may not be sufficient to express some more complicated configuration options possible for some plugins. If that is the case, the map could maintain a structured JSON blob that the plugin would be responsible for parsing. For convenience, plugin writer could provide a “JSON blob creation tool” to make it easier for cluster admins to generate the blob.</p>
<p dir="auto">CC <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a> @kubernetes/goog-cluster @kubernetes/rh-storage</p>
<p dir="auto">We'll use this document to drive the discussion at the Storage Special Interest Group meeting on Dec. 8, 2015 11 AM PST.</p>
<p dir="auto">At github page <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/ubuntu.md">https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/ubuntu.md</a></p>
<p dir="auto">Mentioned one page: <a href="http://kubernetes.github.io/docs/getting-started-guides/ubuntu/" rel="nofollow">http://kubernetes.github.io/docs/getting-started-guides/ubuntu/</a></p>
<p dir="auto">The above link was not found (see attached screenshot)</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/1780032/21073648/773c5d28-bf1f-11e6-9895-f1174bd05864.png"><img src="https://cloud.githubusercontent.com/assets/1780032/21073648/773c5d28-bf1f-11e6-9895-f1174bd05864.png" alt="image003" style="max-width: 100%;"></a></p>