Documenting our migration to Docker
&gt; “Whatever is the underlying executing system, Docker can run it with the exact same code, byte by byte.”<p>Having recently delved into containers, I don&#x27;t like this quote. It&#x27;s not <i>wrong</i>, but I think it&#x27;s also misleading. In particular:<p>&gt; Whatever is the underlying execution system<p>Docker can only execute containers on Linux. Indeed, Docker itself is essentially a UI and management tool for Linux kernel features (chroot, bind mounts, etc.). Whilst Docker provides software for other platforms, like macOS, those are just UIs bundled with a VM&#x2F;hypervisor running Linux; and it&#x27;s that Linux system which runs the containers.<p>This may seem pedantic, but I think it&#x27;s important to understand (a) what&#x27;s actually happening when we run a container, and (b) what we can&#x2F;can&#x27;t do with such tools. For example, I was confused why my containers couldn&#x27;t run macOS binaries: the whole point of containers is that they run inside the host OS, unlike VMs which have their own OS. That&#x27;s certainly true, but I didn&#x27;t realise that when running Docker on macOS, the host is still a VM running Linux!<p>&gt; Docker can run it<p>Again, pedantic but important: Docker is a UI and management tool for containers, which are ultimately run by an underlying Linux system (usually via `runc`, or something compatible). I find this important since &quot;Docker&quot; provides much more than just running containers, e.g. it has Dockerfiles, images, registries, etc.<p>If someone just wants to run a container, they don&#x27;t need Docker; and I&#x27;ve found Docker to be a more complicated and convoluted way of running containers than, say, OCI tools.<p>For example, I recently wrote an AWS Lambda function which needs to run from a custom container (it bundles some tools which are larger than Lambda&#x27;s 50MB code limit; whilst containers have a 10GB limit). I originally did this with Docker, which required:<p>- Building an &#x27;image&#x27; containing the software (I actually did this with Nix, rather than `docker build`, for reasons I&#x27;ll give below)<p>- &quot;Loading&quot; that image into Docker<p>- &quot;Tagging&quot; the image with the URL of an ECR &#x27;image repository&#x27; (we could include this tag during building, but I find this way keeps more distinction between &#x27;building&#x27; and &#x27;deploying&#x27;)<p>- Running `aws ecr get-login | docker login` to &quot;log in&quot; to Docker. This seems ludicrous to me; I hear it&#x27;s something to do with dockerhub compatibility or somesuch; which is still silly, since we&#x27;re not using that.<p>- Pushing to ECR using Docker<p>In contrast, I&#x27;ve now switched that project to use OCI images, which only requires the following:<p>- Building an &#x27;image&#x27; containing the software. This is just a .tar.gz file, plus some .json files which specify the &quot;EntryPoint&quot;, the SHA256 of the .tar.gz, etc. (all easily created with bash + tar + jq)<p>- Uploading the image to ECR. This can be done with `aws ecr` shell commands, but they&#x27;re quite low-level (e.g. files are uploaded 20MB at a time, which needs a loop) so I did this in Python using boto3. Note that the &#x27;tag&#x27; is still needed, but it&#x27;s just an argument to the &#x27;ecr.put_image&#x27; function.<p>&gt; the exact same code, byte by byte<p>This is my main problem with the quote. Whilst it might theoretically be possible to use Docker &quot;properly&quot;, it seems to actively <i>encourage</i> awful practices.<p>For example, to run &quot;the exact same code, byte for byte&quot; we would need to actually <i>get</i> those bytes in the first place. Whilst the underlying container is simply a directory (known as a &quot;bundle&quot;, which `runc` actually executes), Docker abstracts over such bundles: first we tar them up into &quot;layers&quot;, which we then list in a JSON file called an &quot;image&quot;, which we then &quot;push&quot; to an &quot;image repository&quot;. To run a container, we &quot;pull&quot; its image from a repository, which we reference by a &quot;tag&quot; (essentially a filename).<p>The de facto tag is &quot;latest&quot;, which appears in all sorts of documentation and tutorials. Tags may get replaced with new images at any time by another push, which makes the idea of &quot;the exact same code, byte for byte&quot; rather misleading.<p>(Thankfully some repositories, like ECR, allow &#x27;immutable tags&#x27;, which can never be overwritten. Rather than &#x27;latest&#x27;, I use hashes of an image&#x27;s content as its tag; so there&#x27;s never a conflict when I push a new image.)<p>Next, we can ask how those bytes come into existence in the first place. In the world of Docker, we don&#x27;t put files into a .tar.gz; instead, we <i>download an entire Linux distribution</i>, then we run a bunch of non-deterministic package management commands like `apt-get update &amp;&amp; apt-get install -y foo`, resulting in a massive, unreproducible binary blob.<p>Whilst it can be argued that Docker will run &#x27;the exact same code, byte by byte&#x27;, it can also be argued that coin tosses are deterministic due to Newton&#x27;s laws of motion (e.g. a reliable Docker system is illustrated in Figure 1 of <a href="http:&#x2F;&#x2F;statweb.stanford.edu&#x2F;~susan&#x2F;papers&#x2F;headswithJ.pdf" rel="nofollow">http:&#x2F;&#x2F;statweb.stanford.edu&#x2F;~susan&#x2F;papers&#x2F;headswithJ.pdf</a> )