Mayastor: Lightning Fast Storage for Kubernetes
Just want to say for anyone running a clusters of machinese (with kubernetes for example) -- OpenEBS provides one of the most approachable and easily to use solutions for storage out there. It scales from hobby grade to enterprise grade in my opinion, and although I can&#x27;t point you to a 1000+ node cluster I run or anything (maybe OpenEBS can), they offer enterprise editions and support which is usually enough to qualify as the minimum viable definition of &quot;enterprise grade&quot;.<p>OpenEBS has many different drivers[0] (which is a good sign IMO) I only use one of them but am planning a post (that may never actually get written) to expand on all of them if&#x2F;when I can:<p>- Jiva[1] (based on Longhorn[1], iSCSI-based, bring your own disk driver)<p>- cStor[2] (baked by OpenEBS, zfs-based used to be the golden child before MayaStor)<p>- MayaStor (written in rust, iSCSI&#x2F;NVMe-oF based, bring your own lower layer)<p>- zfs-localpv[3]<p>Funnily enough, I&#x27;ve been <i>really</i> interested in zfs and btrfs lately so have been learning a lot about the field, but cStor didn&#x27;t work when I first ran it.<p>I know &quot;written in rust&quot; has become somewhat of a buzzword, but don&#x27;t forget that it has actual meaning -- it means that they will be able to get the raw performance possible without a language runtime or garbage collection to hold them back. This is critical for &quot;systems&quot; programming, and is the domain rust was built for. When I see rust, I basically think -- this could be as fast as average C (and safer) if you get even just pretty good developers.<p>Also a note on this -- one of the things that&#x27;s nice about OpenEBS&#x27;s offerings is that writes are <i>synchronous</i>. This is obviously a problem[4] for performance of course, but it&#x27;s a massive relief for building resiliency into systems. I&#x27;m not a classical sysadmin, but it puts me at ease that I can give up some throughput to know that if one of my servers goes completely kaput, the other can pick up <i>at the last write</i> on the other node, with zero guesswork on my end. The switchover is of course a little hairy but this is what OpenEBS smoothes out for you.<p>For this not to be completely an ad for OpenEBS, I do want to note that the other prominent option that scales from hobbyist to enterprise (and arguably more so) is Rook[5] which manages Rook clusters. Rook the organization seems to be branching out to position themselves as Cassandra storage and a bunch of other stuff, but the key thing they brought to the table was a mostly hands-off operator for operating Ceph[6] which is certified enterprise grade(tm) <i>FREE</i> software. Ceph is basically the linux of the storage appliance world.<p>Ceph (via Rook) is another really good choice for getting storage up and running on a kubernetes (or whatever else) because it offers similar benefits to OpenEBS&#x27;s solutions but does it slightly different. Instead of container-attached-storage, it solves the features come at the  filesystem layer-- you get a tunable &quot;ceph&quot; installation (you should <i>definitely</i> choose BlueStore btw) which gives you:<p>- replication (like the iSCSI approaches of OpenEBS)<p>- checksumming (like ZFS and zfs-backed OpenEBS)[7]<p>- striping (like ZFS&#x2F;btrfs and zfs-backed OpenEBS depending on setup)<p>- compression (like ZFS)[7]<p>Anyway, I don&#x27;t run any servers that use NVMe but nice to see this article, hope the people at OpenEBS are still doing well, love their entry into the space.<p>[EDIT] - I totally forgot and should mention, you can obviously <i>always</i> bring your own lower layer, ex creating a xfs-formatted ZVOL on top of ZFS on the host and then exposing <i>that</i> drive to whatever. Excellent example is a recent talk from Japan which I&#x27;ve mentioned before[8][9] which uses this to run Ceph on ZFS.<p>[0]: <a href="https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;casengines.html" rel="nofollow">https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;casengines.html</a><p>[1]: <a href="https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;jiva.html" rel="nofollow">https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;jiva.html</a><p>[2]: <a href="https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;alphafeatures.html#mayastor" rel="nofollow">https:&#x2F;&#x2F;docs.openebs.io&#x2F;docs&#x2F;next&#x2F;alphafeatures.html#mayasto...</a><p>[3]: <a href="https:&#x2F;&#x2F;github.com&#x2F;openebs&#x2F;zfs-localpv&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openebs&#x2F;zfs-localpv&#x2F;</a><p>[4]: <a href="https:&#x2F;&#x2F;github.com&#x2F;longhorn&#x2F;longhorn&#x2F;issues&#x2F;1242#issuecomment-682066185" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;longhorn&#x2F;longhorn&#x2F;issues&#x2F;1242#issuecommen...</a><p>[5]: <a href="https:&#x2F;&#x2F;rook.io&#x2F;docs&#x2F;rook&#x2F;" rel="nofollow">https:&#x2F;&#x2F;rook.io&#x2F;docs&#x2F;rook&#x2F;</a><p>[6]: <a href="https:&#x2F;&#x2F;docs.ceph.com&#x2F;en&#x2F;latest&#x2F;architecture&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.ceph.com&#x2F;en&#x2F;latest&#x2F;architecture&#x2F;</a><p>[7]: <a href="https:&#x2F;&#x2F;ceph.io&#x2F;community&#x2F;new-luminous-bluestore&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ceph.io&#x2F;community&#x2F;new-luminous-bluestore&#x2F;</a><p>[8]: <a href="https:&#x2F;&#x2F;speakerdeck.com&#x2F;takutakahashi&#x2F;ceph-on-zfs" rel="nofollow">https:&#x2F;&#x2F;speakerdeck.com&#x2F;takutakahashi&#x2F;ceph-on-zfs</a><p>[9]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=24784596" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=24784596</a>