Docker without Docker
I&#x27;m really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack.<p>I aim to build a platform like theirs someday (probably not any time soon) but I don&#x27;t think I&#x27;d do any of what they&#x27;re doing -- it feels unnecessary, because I feel like I know lots of projects that have actually already done the hard work for this. I probably only think this because I haven&#x27;t <i>actually done it</i>, but think it&#x27;s worth sharing the tech anyway. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I&#x27;d love to hear why the following technologies were decided against (if they were):<p>- kata-containers[1] (it does the whole container -&gt; VM flow for you, automatically) with multiple VMM options[2]<p>- linuxkit[3] (let&#x27;s say you didn&#x27;t go with kata-containers, this is another container-&gt;VM path)<p>- firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM)<p>- kubevirt[5] (if you just want to actually run VMs, regardless of how you built them)<p>- Ceph[6] for storage -- make LVM pools and just give them to Ceph, you&#x27;ll get blocks, distributed filesystems (CephFS), and object gateways (S3&#x2F;Swift) out of it (in the k8s space Rook manages this)<p>As an aside to all this, there&#x27;s also LXD, which supports running &quot;system&quot; (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management&#x2F;migration of underlying filesystems, runs on LVM or zfs[10], it&#x27;s basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the &quot;cloud native&quot;&#x2F;&quot;works-with-kubernetes&quot; train.<p>I&#x27;ve basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and&#x2F;or what was turned down.<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26745514" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26745514</a><p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;kata-containers&#x2F;kata-containers" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kata-containers&#x2F;kata-containers</a><p>[2]: <a href="https:&#x2F;&#x2F;github.com&#x2F;kata-containers&#x2F;kata-containers&#x2F;blob&#x2F;2fc7f75724ac9e18e60f63dcc9aa395dc51c184d&#x2F;docs&#x2F;design&#x2F;virtualization.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kata-containers&#x2F;kata-containers&#x2F;blob&#x2F;2fc7...</a><p>[3]: <a href="https:&#x2F;&#x2F;github.com&#x2F;linuxkit&#x2F;linuxkit" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;linuxkit&#x2F;linuxkit</a><p>[4]: <a href="https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker-containerd&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker-container...</a><p>[5]: <a href="https:&#x2F;&#x2F;github.com&#x2F;kubevirt&#x2F;kubevirt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kubevirt&#x2F;kubevirt</a><p>[6]: <a href="https:&#x2F;&#x2F;docs.ceph.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.ceph.com&#x2F;</a><p>[7]: <a href="https:&#x2F;&#x2F;discuss.linuxcontainers.org&#x2F;t&#x2F;running-virtual-machines-with-lxd-4-0&#x2F;7519" rel="nofollow">https:&#x2F;&#x2F;discuss.linuxcontainers.org&#x2F;t&#x2F;running-virtual-machin...</a><p>[8]: <a href="https:&#x2F;&#x2F;github.com&#x2F;lxc&#x2F;lxd&#x2F;issues&#x2F;6205" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;lxc&#x2F;lxd&#x2F;issues&#x2F;6205</a><p>[9]: <a href="https:&#x2F;&#x2F;criu.org&#x2F;Main_Page" rel="nofollow">https:&#x2F;&#x2F;criu.org&#x2F;Main_Page</a><p>[10]: <a href="https:&#x2F;&#x2F;linuxcontainers.org&#x2F;lxd&#x2F;docs&#x2F;master&#x2F;storage" rel="nofollow">https:&#x2F;&#x2F;linuxcontainers.org&#x2F;lxd&#x2F;docs&#x2F;master&#x2F;storage</a>