Answers: <blockquote>
  <p>I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.</p>
  </blockquote>
  <p>In reality, it is highly unlikely that any development environment on any project would ever be anywhere near the same as staging/production.</p>
  <ul>
  <li>Services running in staging/production will nearly always be physically hosted and managed somewhere which is not intended to be operated interactively by a human day-to-day, with an appropriate IT/security profile to match;</li>
  <li>The nature of development work, and even internal build/testing typically requires a different IT profile to that of a production server.</li>
  <li>Developers rarely have control over the underlying infrastructure or the organisation's IT/security policies.</li>
  </ul>
  <p>There are many ways in which the IT profile of developer environments, including build agents and even test machines, can deviate from production:</p>
  <ul>
  <li>Users/permissions or other security settings.</li>
  <li>Installed tools, SDKs, runtimes, debug/test tools, OS features/packages, and other dependencies operating with debug/test configurations enabled.</li>
  <li>Environment variables</li>
  <li>Filesystem structure and the content of files in globally shared directories</li>
  <li>Configurations of globally-installed dependencies such as web servers.</li>
  </ul>
  <p>Furthermore, consider the nature of physical devices and VMs</p>
  <ul>
  <li>They are stateful and mutable</li>
  <li>Every change to any aspect of a device or VM, including installed software and configuration changes potentially affects its entire state for all processes running on it.</li>
  <li>Physical devices and VMs typically run many processes/services concurrently, it would usually not be considered economical to have a whole server or VM just for a single running process.</li>
  </ul>
  <p>What containers provide:</p>
  <ul>
  <li>Isolation from the host device/VM and from an organisation's IT, Network and Infrastructure policies.</li>
  <li>Isolation from each other - for example, consider the issue of requiring multiple versions of globally-installed runtime dependencies or modifications to shared host resources such as environment variables or local files.</li>
  <li>Developers typically have full control over their choices of container images and the networking/orchestration inside the container runtime.</li>
  <li>Images are based on immutable layers, meaning it is not possible for the state of any layer in an image to change, so a published image should always be a good, known, valid starting point.</li>
  <li>The size of a parent image tends to be inconsequential because there's typically no reason to duplicate nor to re-download it unless a new version of that parent image is published.</li>
  <li>A container is its own thin, mutable layer on top of an image, usually negligible in size, and uses the parent image for all dependencies.</li>
  <li>if a container ends up in an invalid state, it may be quickly and cheaply disposed and replaced with a fresh, clean new container almost instantly by recreating another fresh new container layer.</li>
  <li>The cheap, light-weight nature of containers makes it very efficient to run a single process per-container.</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-31T13:58:02.557Z
Answers: Ben, thanks a lot for this detailed answer. I appreciate the higher-level approach to the answer. I can appreciate the flexibility that containers provide for DevOps teams, especially in the enterprise environment you have described. Without dismissing Docker, I'm looking into building [in the open] an opinionated alternative to it (perhaps leveraging LXD) for a low-code platform [currently idea stage]; 5Mb would be the limit for container images (any larger would be from app dependencies). Could you share any resources, books, etc. that would help me boost knowledge on this?
Answers: ObiHill
Answers: CC BY-SA 4.0
Answers: 2022-08-31T14:14:53.593Z
Answers: Also happy to have a quick chat if possible :)
Answers: ObiHill
Answers: CC BY-SA 4.0
Answers: 2022-08-31T17:25:45.823Z
Answers: @ObiHill The underlying Linux Kernel feature (for Docker and I assume for LXD too) is Linux Namespaces, so you could consider starting there to get an understanding of those - the best place is likely to be in the official docs for some popular distro such as Red Hat.   -- Docker itself is already really quite a thin layer of abstraction around namespaces (IMO), so while you could certainly roll your own using namespaces (and I obviously do not know your specific requirements), you may just end up reinventing docker without all of its useful and convenient features to help you manage it.
Answers: Ben Cottrell
Answers: CC BY-SA 4.0
Answers: 2022-08-31T17:31:37.573Z
Answers: Also obviously Docker's own docs are very well worth any time spent and its guides/tutorials for getting some simple containers working to learn its tools and capabilities (if nothing else, to understand the kind of stuff you'd be missing out and would very likely need to reinvent if you decided to create your own, at least if you wanted any chance of being able to support and maintain it in production -- e.g. the docker CLI, its logging, configuration model, various convenient ways to observe a running container and diagnose problems.)
Answers: Ben Cottrell
Answers: CC BY-SA 4.0
Answers: 2022-08-31T17:38:48.807Z
Answers: Also I think Docker in any system/project is usually just one in a much longer list of tools -- Docker is handy for the one very narrow and specific set of problems it solves (A tool which does one thing and does it well), but it still leaves you with plenty of other potentially tricky issues to consider -- https://12factor.net/  (At least, I think these factors are all worth considering for anyone looking at a development platform because they tend to play a large role in how people support and maintain a production system over many years)
Answers: Ben Cottrell
Answers: CC BY-SA 4.0
Answers: 2022-09-03T07:00:45.380Z
Answers: Thanks Ben. 
  
  (1) As regards my specific requirements, the low-code tool will generate workloads that need to run in the cloud. Containerization appears to be the optimal approach for this (i.e. via CaaS platforms like AWS Fargate, GCP Cloud Run, etc.). In addition, having micro-VMs (like Firecracker) as a deployment option enhances the overall proposition. My focus is the most secure container/VM possible at the lowest possible image size. However, I'm torn on whether to leverage Docker or build a minimal solution from scratch. Will look at Linux namespaces and explore what's possible.
Answers: ObiHill
Answers: CC BY-SA 4.0
Answers: 2022-09-03T07:13:10.927Z
Answers: (2) It appears regardless of what decision is taken (use Docker or build an alternative), intermediate to advanced Docker knowledge will be useful for the deployment and orchestration tooling of the overall low-code solution i.e. to enable someone that already has a workload (container or VM) to deploy/manage/monitor it, rather than forcing them to recreate it using the planned low-code GUI. So, your points are well taken and the Docker Docs and a few highly-regarded books will be my starting point.
Answers: ObiHill
Answers: CC BY-SA 4.0
Answers: 2022-09-03T10:36:22.790Z
Answers: (3) I'm familiar with 12factor.net. I also recently stumbled on [Beyond the 12 factor app](https://tanzu.vmware.com/content/blog/beyond-the-twelve-factor-app) which adds 3 more factors to the mix. I'll be considering them all in the process going forward.
Answers: ObiHill
Answers: CC BY-SA 4.0
Answers: 2022-08-31T04:54:57.423Z
Answers: 2022-08-31T05:01:25.143Z
Answers: 2022-08-31T05:01:25.143Z
Answers: Ben Cottrell
Body: <p>I'm trying to get a better understanding of the reasons to use [and not use] Docker based on specific use cases.</p>
  <p>From my current understanding, Docker helps to isolate applications and their dependencies within containers. This is useful to ensure consistent reproducible builds in varied environments.</p>
  <p>However, I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.</p>
  <p>Say I have the following:</p>
  <ul>
  <li>a cloud VM instance (DigitalOcean, Vultr, Linode, etc.) with 1Gb RAM running Ubuntu 20.</li>
  <li>a Node.js Express app (nothing too complicated)</li>
  </ul>
  <p>The following issues come to the fore:</p>
  <ol>
  <li><p><em>Dockerizing</em> this application will produce an image that is ~100Mb after optimization (without optimization probably 500Mb or higher based on my research). The app could be 50Kb in size, but the Docker container dependencies to run it are significantly higher by a factor of up to 10,000 or above. This seems very unreasonable from an optimization standpoint.</p>
  </li>
  <li><p>I have to push this container image to a hub before I can use Docker to consume it. So that's 500Mb to the hub, and then 500Mb down to my VM instance; total of about 1Gb of bandwidth per build. Multiply this by the number of times the build needs to be updated and you could be approaching terabytes in bandwidth usage.</p>
  </li>
  <li><p>I read in a <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-22-04" rel="nofollow noreferrer">DigitalOcean tutorial</a> that before I can run my container image, I have to do the following:</p>
  </li>
  </ol>
  <pre><code>docker pull ubuntu
  </code></pre>
  <p>This pulls an image of Ubuntu. But, I'm already on Ubuntu, so does this mean I'm running a container that's running Ubuntu inside an existing VM that is running Ubuntu? This appears to be needless duplication, but I'd appreciate clarification.</p>
  <ol start="4">
  <li>The <a href="https://docs.docker.com/desktop/install/linux-install/" rel="nofollow noreferrer">Docker Docs</a> specify that I should have 4Gb RAM. This means I have to use more expensive VM instances even when my application does not necessarily require it.</li>
  </ol>
  <p>How exactly does <em>containerization</em> [using Docker or similar] optimize and enhance the DevOps experience, especially on an ongoing basis?</p>
  <p>I'm not quite <em>getting it</em> but I'm open to clarification.</p>
  
Comments: CC BY-SA 4.0
Comments: 2022-08-30T14:43:23.187Z
Comments: After you Dockerize things you no longer need VM. All you need is a service that runs the container for you. Of course you can still use a VM and handle the "orchestration" yourself. For this simple case you really don't need a container, unless you gonna run it in ECS (or such) and save on the DO droplet (or if you are lone developer). As for the "app could be 50Kb in size", yea but you need all dependencies (mind you with correct versions) on the host machine, with containers (Docker) you can leverage layers / caching or run the same app with different version of dependencies.
Comments: Kyslik
Comments: CC BY-SA 4.0
Comments: 2022-08-31T14:13:43.007Z
Comments: @Kyslik Thanks for the answer. So with this "service that runs the container for you", they're still running virtualized hardware? Also, I now see the benefit of being able to run multiple versions of the same software on different containers for compatibility between different apps. Still, if you're doing that it might mean that you're spreading yourself too thin and should probably consolidate the applications you use i.e. [be more boring](https://mcfunley.com/choose-boring-technology).
Comments: ObiHill
Tags: <docker><containers>
Title: Rationale for using Docker to containerize applications
UserDisplayName: ObiHill
ThreadText: # Rationale for using Docker to containerize applications
  
  (Asked by: username_0 on 2022-08-30)
  
  I'm trying to get a better understanding of the reasons to use [and not use] Docker based on specific use cases.
  
  From my current understanding, Docker helps to isolate applications and their dependencies within containers. This is useful to ensure consistent reproducible builds in varied environments.
  
  However, I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.
  
  Say I have the following:
  
  * a cloud VM instance (DigitalOcean, Vultr, Linode, etc.) with 1Gb RAM running Ubuntu 20.
  * a Node.js Express app (nothing too complicated)
  
  The following issues come to the fore:
  
  1. *Dockerizing* this application will produce an image that is ~100Mb after optimization (without optimization probably 500Mb or higher based on my research). The app could be 50Kb in size, but the Docker container dependencies to run it are significantly higher by a factor of up to 10,000 or above. This seems very unreasonable from an optimization standpoint.
  2. I have to push this container image to a hub before I can use Docker to consume it. So that's 500Mb to the hub, and then 500Mb down to my VM instance; total of about 1Gb of bandwidth per build. Multiply this by the number of times the build needs to be updated and you could be approaching terabytes in bandwidth usage.
  3. I read in a [DigitalOcean tutorial](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-22-04) that before I can run my container image, I have to do the following:
  
  ```
  docker pull ubuntu
  
  ```
  
  This pulls an image of Ubuntu. But, I'm already on Ubuntu, so does this mean I'm running a container that's running Ubuntu inside an existing VM that is running Ubuntu? This appears to be needless duplication, but I'd appreciate clarification.
  
  4. The [Docker Docs](https://docs.docker.com/desktop/install/linux-install/) specify that I should have 4Gb RAM. This means I have to use more expensive VM instances even when my application does not necessarily require it.
  
  How exactly does *containerization* [using Docker or similar] optimize and enhance the DevOps experience, especially on an ongoing basis?
  
  I'm not quite *getting it* but I'm open to clarification.
  
  **Tags:** docker, containers
  
  > **Comments:**
  > **username_1**: After you Dockerize things you no longer need VM. All you need is a service that runs the container for you. Of course you can still use a VM and handle the "orchestration" yourself. For this simple case you really don't need a container, unless you gonna run it in ECS (or such) and save on the DO droplet (or if you are lone developer). As for the "app could be 50Kb in size", yea but you need all dependencies (mind you with correct versions) on the host machine, with containers (Docker) you can leverage layers / caching or run the same app with different version of dependencies.  
  > **username_0**: @username_1 Thanks for the answer. So with this "service that runs the container for you", they're still running virtualized hardware? Also, I now see the benefit of being able to run multiple versions of the same software on different containers for compatibility between different apps. Still, if you're doing that it might mean that you're spreading yourself too thin and should probably consolidate the applications you use i.e. [be more boring](https://mcfunley.com/choose-boring-technology).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-31. Score: 2 | Accepted answer
  
  > I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.
  
  In reality, it is highly unlikely that any development environment on any project would ever be anywhere near the same as staging/production.
  
  * Services running in staging/production will nearly always be physically hosted and managed somewhere which is not intended to be operated interactively by a human day-to-day, with an appropriate IT/security profile to match;
  * The nature of development work, and even internal build/testing typically requires a different IT profile to that of a production server.
  * Developers rarely have control over the underlying infrastructure or the organisation's IT/security policies.
  
  There are many ways in which the IT profile of developer environments, including build agents and even test machines, can deviate from production:
  
  * Users/permissions or other security settings.
  * Installed tools, SDKs, runtimes, debug/test tools, OS features/packages, and other dependencies operating with debug/test configurations enabled.
  * Environment variables
  * Filesystem structure and the content of files in globally shared directories
  * Configurations of globally-installed dependencies such as web servers.
  
  Furthermore, consider the nature of physical devices and VMs
  
  * They are stateful and mutable
  * Every change to any aspect of a device or VM, including installed software and configuration changes potentially affects its entire state for all processes running on it.
  * Physical devices and VMs typically run many processes/services concurrently, it would usually not be considered economical to have a whole server or VM just for a single running process.
  
  What containers provide:
  
  * Isolation from the host device/VM and from an organisation's IT, Network and Infrastructure policies.
  * Isolation from each other - for example, consider the issue of requiring multiple versions of globally-installed runtime dependencies or modifications to shared host resources such as environment variables or local files.
  * Developers typically have full control over their choices of container images and the networking/orchestration inside the container runtime.
  * Images are based on immutable layers, meaning it is not possible for the state of any layer in an image to change, so a published image should always be a good, known, valid starting point.
  * The size of a parent image tends to be inconsequential because there's typically no reason to duplicate nor to re-download it unless a new version of that parent image is published.
  * A container is its own thin, mutable layer on top of an image, usually negligible in size, and uses the parent image for all dependencies.
  * if a container ends up in an invalid state, it may be quickly and cheaply disposed and replaced with a fresh, clean new container almost instantly by recreating another fresh new container layer.
  * The cheap, light-weight nature of containers makes it very efficient to run a single process per-container.
  
  > **Comments:**
  > **username_0**: Ben, thanks a lot for this detailed answer. I appreciate the higher-level approach to the answer. I can appreciate the flexibility that containers provide for DevOps teams, especially in the enterprise environment you have described. Without dismissing Docker, I'm looking into building [in the open] an opinionated alternative to it (perhaps leveraging LXD) for a low-code platform [currently idea stage]; 5Mb would be the limit for container images (any larger would be from app dependencies). Could you share any resources, books, etc. that would help me boost knowledge on this?  
  > **username_0**: Also happy to have a quick chat if possible :)  
  > **username_2**: @username_0 The underlying Linux Kernel feature (for Docker and I assume for LXD too) is Linux Namespaces, so you could consider starting there to get an understanding of those - the best place is likely to be in the official docs for some popular distro such as Red Hat.   -- Docker itself is already really quite a thin layer of abstraction around namespaces (IMO), so while you could certainly roll your own using namespaces (and I obviously do not know your specific requirements), you may just end up reinventing docker without all of its useful and convenient features to help you manage it.  
  > **username_2**: Also obviously Docker's own docs are very well worth any time spent and its guides/tutorials for getting some simple containers working to learn its tools and capabilities (if nothing else, to understand the kind of stuff you'd be missing out and would very likely need to reinvent if you decided to create your own, at least if you wanted any chance of being able to support and maintain it in production -- e.g. the docker CLI, its logging, configuration model, various convenient ways to observe a running container and diagnose problems.)  
  > **username_2**: Also I think Docker in any system/project is usually just one in a much longer list of tools -- Docker is handy for the one very narrow and specific set of problems it solves (A tool which does one thing and does it well), but it still leaves you with plenty of other potentially tricky issues to consider -- https://12factor.net/  (At least, I think these factors are all worth considering for anyone looking at a development platform because they tend to play a large role in how people support and maintain a production system over many years)  
  > **username_0**: Thanks Ben.   (1) As regards my specific requirements, the low-code tool will generate workloads that need to run in the cloud. Containerization appears to be the optimal approach for this (i.e. via CaaS platforms like AWS Fargate, GCP Cloud Run, etc.). In addition, having micro-VMs (like Firecracker) as a deployment option enhances the overall proposition. My focus is the most secure container/VM possible at the lowest possible image size. However, I'm torn on whether to leverage Docker or build a minimal solution from scratch. Will look at Linux namespaces and explore what's possible.  
  > **username_0**: (2) It appears regardless of what decision is taken (use Docker or build an alternative), intermediate to advanced Docker knowledge will be useful for the deployment and orchestration tooling of the overall low-code solution i.e. to enable someone that already has a workload (container or VM) to deploy/manage/monitor it, rather than forcing them to recreate it using the planned low-code GUI. So, your points are well taken and the Docker Docs and a few highly-regarded books will be my starting point.  
  > **username_0**: (3) I'm familiar with 12factor.net. I also recently stumbled on [Beyond the 12 factor app](https://tanzu.vmware.com/content/blog/beyond-the-twelve-factor-app) which adds 3 more factors to the mix. I'll be considering them all in the process going forward.  
  
  ---
  
  