Answers: <p><strong>Docker is still in development.</strong></p>
  
  <p>As with every other software in-dev bugs will happen, insecure features might be added, there could be architectural flaws that lead to security breaches. Do not underestimate this! Your system might be completely safe today, but with the patch from next week someone finds a bug, writes an exploit, and suddenly your system is wide open.</p>
  
  <p>Unless you must, don't update to the latest version. Use the latest well-tested version instead.</p>
  
  <p><strong>Docker is no virtualization</strong></p>
  
  <p>If someone escapes from a Docker container, that attacker is on the real machine immediately. There is no second gate like virtualization that will prevent a breach.</p>
  
  <p>Treat a Docker container like any other program. Run with lowest possible user rights, block all network traffic that isn't required, virtualize the entire Docker host if performance allows.</p>
  
  <p><strong>Docker is no protection</strong></p>
  
  <p>Whatever code is run inside Docker containers is run without question from Docker. Any attacker can simply install his software inside the container, and Docker would run that like any other code.</p>
  
  <p>Aside from the things you mentioned in the question, consider using metrics and alerts to get notified if any Docker image is doing strange things. Is there a sudden, ongoing CPU spike? Is the program suddenly scanning network ports? Is there suspicious disk access? You should get a notification if any of that happens. There are many tools available to measure these things, you should use them.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-07T12:42:20.963Z
Answers: 2017-09-07T14:54:01.743Z
Answers: 2017-09-07T14:54:01.743Z
Answers: TwoThe
Answers: <p>If you are filling up your docker entrypoint with <code>sed</code> commands, consider this practice:</p>
  
  <ul>
  <li>Use a tool such as <a href="https://github.com/kelseyhightower/confd" rel="nofollow noreferrer">confd</a> to manage your docker images configuration files and keep them updated</li>
  </ul>
  
  <p><a href="https://github.com/kelseyhightower/confd" rel="nofollow noreferrer">Confd</a> will read data from many supported key-value stores and render configuration templates dynamically.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-27T22:08:15.477Z
Answers: 2018-02-27T22:08:15.477Z
Answers: Vincenzo Pii
Answers: <p><strong>Docker images itself</strong></p>
  
  <p>An additional option is to use <a href="https://github.com/coreos/clair" rel="noreferrer">Clair</a>.</p>
  
  <blockquote>
    <p>Clair is an open source project for the static analysis of
    vulnerabilities in application containers (currently including appc
    and docker).</p>
    
    <p>In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database.</p>
    
    <p>Clients use the Clair API to index their container images; this creates a list of features present in the image and stores them in the
    database.</p>
    
    <p>Clients use the Clair API to query the database for vulnerabilities of a particular image; correlating vulnerabilities and
    features is done for each request, avoiding the need to rescan images.</p>
    
    <p>When updates to vulnerability metadata occur, a notification can be sent to alert systems that a change has occured.</p>
    
    <p>Our goal is to enable a more transparent view of the security of
    container-based infrastructure. Thus, the project was named Clair
    after the French term which translates to clear, bright, transparent.</p>
  </blockquote>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-07T09:28:11.800Z
Answers: 2017-09-07T21:21:10.507Z
Answers: 2017-09-07T21:21:10.507Z
Answers: 030
Answers: <p>In addition to the points in this thread; the following would be my recommendation:</p>
  
  <ul>
  <li>Get control over Docker PID1 with dumb-init
  
  <ul>
  <li>ref: <a href="https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/" rel="noreferrer">PID1 and Zombie reaping problem</a></li>
  </ul></li>
  <li>Do not run docker in production without a container orchestration system
  
  <ul>
  <li>Take your pick from Kubernetes, Mesos, Swarm etc.</li>
  </ul></li>
  <li>Use <a href="https://github.com/tianon/gosu" rel="noreferrer">gosu</a> for user control inside a docker image</li>
  <li>Follow the 12 factor app paradigm, if you are running stateful apps in containers, change it up.
  
  <ul>
  <li>If you really need to run stateful apps (mysql, zookeeper, elasticsearch) in containers, leverage orchestrator paradigms like <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="noreferrer">Kubernetes Statefulsets</a></li>
  </ul></li>
  <li>Do robust secret / config management with tools like hashicorp vault / consul</li>
  <li>Ship the same container built by the devs to prod through a CI pipeline that takes it through staging, integration-tests thoroughly.</li>
  <li>Create notifications around CVEs and patches, trigger builds on patch-notify</li>
  <li>Have extensive logging to get insight into the running container, you do not want to give the devs SSH access to either the host or the containers
  
  <ul>
  <li>recommendation: fluentd</li>
  </ul></li>
  <li>Have both container and host metrics
  
  <ul>
  <li>recommendation: prometheus+node-exporter</li>
  </ul></li>
  </ul>
  
Answers: CC BY-SA 3.0
Answers: 2017-11-29T20:48:45.093Z
Answers: 2017-11-29T20:48:45.093Z
Answers: Hashfyre
Answers: <p>One could use <a href="https://github.com/030/a2d" rel="nofollow noreferrer">A2D</a> to bake an app into a docker image while taking certain things into account, e.g. non-root, permissions, location of the app:</p>
  
  <blockquote>
  <pre><code>docker run -v $PWD:/projectName utrecht/a2d:1.0.0 \
         -projectName someProjectName -dockerfile /projectName/Dockerfile
  </code></pre>
  </blockquote>
  
  <p>returns:</p>
  
  <blockquote>
  <pre><code>FROM golang:1.12.4-alpine as builder
  COPY . ./someProjectName/
  WORKDIR someProjectName
  RUN adduser -D -g '' someProjectName &amp;&amp; \
      apk add git &amp;&amp; \
      CGO_ENABLED=0 go build &amp;&amp; \
      cp someProjectName /someProjectName &amp;&amp; \
      chmod 100 /someProjectName
  
  FROM scratch
  COPY --from=builder /etc/group /etc/group
  COPY --from=builder /etc/passwd /etc/passwd
  COPY --from=builder --chown=someProjectName:someProjectName /someProjectName /usr/local/someProjectName
  COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
  USER someProjectName
  ENTRYPOINT ["/usr/local/someProjectName"]
  </code></pre>
  </blockquote>
  
Answers: CC BY-SA 4.0
Answers: 2019-05-13T22:05:07.950Z
Answers: 2019-05-13T22:05:07.950Z
Answers: 030
Answers: <p>Reading <a href="https://docs.docker.com/engine/security/security/" rel="nofollow noreferrer">the official docker security article</a> and comparing it with the bullets that are defined in the question the following things are important as well:</p>
  
  <ul>
  <li>protect the docker daemon socket by implementing ssl</li>
  <li>content trust using notary and <code>DOCKER_CONTENT_TRUST</code> variable</li>
  </ul>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-07T14:56:08.833Z
Answers: You should try to compile your answer, this part could be within any of your other two answers
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-09-07T10:27:48.933Z
Answers: 2017-09-07T10:27:48.933Z
Answers: 030
Answers: <p><strong>The host on which the containers are running</strong></p>
  
  <p>Run the docker security bench on every node that runs docker containers <a href="https://github.com/docker/docker-bench-security" rel="noreferrer">https://github.com/docker/docker-bench-security</a></p>
  
  <p>Running the following command on a node that runs docker containers:</p>
  
  <pre><code>docker run -it --net host --pid host --cap-add audit_control \
      -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
      -v /var/lib:/var/lib \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v /usr/lib/systemd:/usr/lib/systemd \
      -v /etc:/etc --label docker_bench_security \
      docker/docker-bench-security
  </code></pre>
  
  <p>returns a list of checks:</p>
  
  <pre><code>[INFO] 1 - Host Configuration
  
  [WARN] 1.1  - Ensure a separate partition for containers has been created
  
  [NOTE] 4.2  - Ensure that containers use trusted base images
  
  [PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image
  </code></pre>
  
  <p>Quote from the repository README:</p>
  
  <blockquote>
    <p>The Docker Bench for Security is a script that checks for dozens of
    common best-practices around deploying Docker containers in
    production. The tests are all automated, and are inspired by the <a href="https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_Community_Edition_Benchmark_v1.1.0.pdf" rel="noreferrer">CIS
    Docker Community Edition Benchmark
    v1.1.0</a>.</p>
  </blockquote>
  
  <p>Some of the issues that are reported by the security bench could be solved by reading <a href="https://docs.docker.com/engine/security/security/" rel="noreferrer">the official docker security article</a> and comparing it with the bullets that are defined in the question the following things are important as well:</p>
  
  <ul>
  <li>protect the docker daemon socket by implementing ssl</li>
  <li>content trust using notary and <code>DOCKER_CONTENT_TRUST</code> variable</li>
  </ul>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-07T08:12:33.183Z
Answers: 2017-09-07T15:06:31.313Z
Answers: 2017-09-07T15:06:31.313Z
Answers: 030
Body: <p>Finally, you are so much in love with Docker that you want to move your online business-critical production systems with sensitive customer data to a Docker Swarm. Some might even already have done so. The other organization can't afford it by a policy forbidding production processes running in root mode.</p>
  
  <p>What could be a checklist of building blocks to consider for a Docker production environment? One does not need all of them, but all of them should be important to be assessed.</p>
  
  <p><em>Disclaimer: I know there is a SE policy to avoid "large endless lists" but I think this checklist cannot be very big... and endless noway.</em></p>
  
  <p>So - what are these buildings blocks?</p>
  
  <ol>
  <li>If not already deployed, consider running a Linux host system with advanced 
  security settings - hardened kernel, SELinux etc.</li>
  <li>Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image</li>
  <li>Use USER setting other than root </li>
  <li>Carefully assess to further reduce the already shrinked set of kernel capabilities granted to container</li>
  <li>Consider having only one executable binary per container to launch your process, ideally statically linked</li>
  <li>Those who want to break your system to get a shell access might wonder if they found out your container has all shells disabled</li>
  <li>Mount read-only volumes where only possible</li>
  </ol>
  
  <p><strong>Question: what else?</strong></p>
  
Comments: CC BY-SA 3.0
Comments: 2017-09-07T08:13:06.577Z
Comments: What does the tag `devsecops` mean?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-07T08:14:01.430Z
Comments: Ok interesting - http://www.devsecops.org/blog/2015/2/15/what-is-devsecops
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-07T09:30:56.553Z
Comments: Could you explain why this `Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image` enhances security?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-07T10:48:07.673Z
Comments: @030 the less  you have installed, the better you can protect against un-needed services/software  that are under-maintained  and/or potentially exploitable. Stripping down to the bare minimum will always work better since each container is supposed to be used to serve a single service./goal.
Comments: Leon
Comments: CC BY-SA 4.0
Comments: 2019-06-30T09:49:08.353Z
Comments: I have answered such similar question here https://stackoverflow.com/a/56643545/6790948 Have a look.
Comments: mchawre
Tags: <docker><security><devsecops><a2d>
Title: What are best and comprehensive practices to consider when running docker in production?
UserDisplayName: Ta Mu
ThreadText: # What are best and comprehensive practices to consider when running docker in production?
  
  (Asked by: username_0 on 2017-09-07)
  
  Finally, you are so much in love with Docker that you want to move your online business-critical production systems with sensitive customer data to a Docker Swarm. Some might even already have done so. The other organization can't afford it by a policy forbidding production processes running in root mode.
  
  What could be a checklist of building blocks to consider for a Docker production environment? One does not need all of them, but all of them should be important to be assessed.
  
  *Disclaimer: I know there is a SE policy to avoid "large endless lists" but I think this checklist cannot be very big... and endless noway.*
  
  So - what are these buildings blocks?
  
  1. If not already deployed, consider running a Linux host system with advanced
     security settings - hardened kernel, SELinux etc.
  2. Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image
  3. Use USER setting other than root
  4. Carefully assess to further reduce the already shrinked set of kernel capabilities granted to container
  5. Consider having only one executable binary per container to launch your process, ideally statically linked
  6. Those who want to break your system to get a shell access might wonder if they found out your container has all shells disabled
  7. Mount read-only volumes where only possible
  
  **Question: what else?**
  
  **Tags:** docker, security, devsecops, a2d
  
  > **Comments:**
  > **username_1**: What does the tag `devsecops` mean?  
  > **username_1**: Ok interesting - http://www.devsecops.org/blog/2015/2/15/what-is-devsecops  
  > **username_1**: Could you explain why this `Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image` enhances security?  
  > **username_2**: @username_1 the less  you have installed, the better you can protect against un-needed services/software  that are under-maintained  and/or potentially exploitable. Stripping down to the bare minimum will always work better since each container is supposed to be used to serve a single service./goal.  
  > **username_3**: I have answered such similar question here https://stackoverflow.com/a/56643545/6790948 Have a look.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_1 on 2017-09-07. Score: 24 | Accepted answer
  
  **The host on which the containers are running**
  
  Run the docker security bench on every node that runs docker containers <https://github.com/docker/docker-bench-security>
  
  Running the following command on a node that runs docker containers:
  
  ```
  docker run -it --net host --pid host --cap-add audit_control \
      -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
      -v /var/lib:/var/lib \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v /usr/lib/systemd:/usr/lib/systemd \
      -v /etc:/etc --label docker_bench_security \
      docker/docker-bench-security
  
  ```
  
  returns a list of checks:
  
  ```
  [INFO] 1 - Host Configuration
  
  [WARN] 1.1  - Ensure a separate partition for containers has been created
  
  [NOTE] 4.2  - Ensure that containers use trusted base images
  
  [PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image
  
  ```
  
  Quote from the repository README:
  
  > The Docker Bench for Security is a script that checks for dozens of
  > common best-practices around deploying Docker containers in
  > production. The tests are all automated, and are inspired by the [CIS
  > Docker Community Edition Benchmark
  > v1.1.0](https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_Community_Edition_Benchmark_v1.1.0.pdf).
  
  Some of the issues that are reported by the security bench could be solved by reading [the official docker security article](https://docs.docker.com/engine/security/security/) and comparing it with the bullets that are defined in the question the following things are important as well:
  
  * protect the docker daemon socket by implementing ssl
  * content trust using notary and `DOCKER_CONTENT_TRUST` variable
  
  
  ---
  
  ### Answer by username_1 on 2017-09-07. Score: 8
  
  **Docker images itself**
  
  An additional option is to use [Clair](https://github.com/coreos/clair).
  
  > Clair is an open source project for the static analysis of
  > vulnerabilities in application containers (currently including appc
  > and docker).
  >
  > In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database.
  >
  > Clients use the Clair API to index their container images; this creates a list of features present in the image and stores them in the
  > database.
  >
  > Clients use the Clair API to query the database for vulnerabilities of a particular image; correlating vulnerabilities and
  > features is done for each request, avoiding the need to rescan images.
  >
  > When updates to vulnerability metadata occur, a notification can be sent to alert systems that a change has occured.
  >
  > Our goal is to enable a more transparent view of the security of
  > container-based infrastructure. Thus, the project was named Clair
  > after the French term which translates to clear, bright, transparent.
  
  
  ---
  
  ### Answer by username_6 on 2017-11-29. Score: 8
  
  In addition to the points in this thread; the following would be my recommendation:
  
  * Get control over Docker PID1 with dumb-init
    + ref: [PID1 and Zombie reaping problem](https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/)
  * Do not run docker in production without a container orchestration system
    + Take your pick from Kubernetes, Mesos, Swarm etc.
  * Use [gosu](https://github.com/tianon/gosu) for user control inside a docker image
  * Follow the 12 factor app paradigm, if you are running stateful apps in containers, change it up.
    + If you really need to run stateful apps (mysql, zookeeper, elasticsearch) in containers, leverage orchestrator paradigms like [Kubernetes Statefulsets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)
  * Do robust secret / config management with tools like hashicorp vault / consul
  * Ship the same container built by the devs to prod through a CI pipeline that takes it through staging, integration-tests thoroughly.
  * Create notifications around CVEs and patches, trigger builds on patch-notify
  * Have extensive logging to get insight into the running container, you do not want to give the devs SSH access to either the host or the containers
    + recommendation: fluentd
  * Have both container and host metrics
    + recommendation: prometheus+node-exporter
  
  
  ---
  
  ### Answer by username_4 on 2017-09-07. Score: 7
  
  **Docker is still in development.**
  
  As with every other software in-dev bugs will happen, insecure features might be added, there could be architectural flaws that lead to security breaches. Do not underestimate this! Your system might be completely safe today, but with the patch from next week someone finds a bug, writes an exploit, and suddenly your system is wide open.
  
  Unless you must, don't update to the latest version. Use the latest well-tested version instead.
  
  **Docker is no virtualization**
  
  If someone escapes from a Docker container, that attacker is on the real machine immediately. There is no second gate like virtualization that will prevent a breach.
  
  Treat a Docker container like any other program. Run with lowest possible user rights, block all network traffic that isn't required, virtualize the entire Docker host if performance allows.
  
  **Docker is no protection**
  
  Whatever code is run inside Docker containers is run without question from Docker. Any attacker can simply install his software inside the container, and Docker would run that like any other code.
  
  Aside from the things you mentioned in the question, consider using metrics and alerts to get notified if any Docker image is doing strange things. Is there a sudden, ongoing CPU spike? Is the program suddenly scanning network ports? Is there suspicious disk access? You should get a notification if any of that happens. There are many tools available to measure these things, you should use them.
  
  
  ---
  
  ### Answer by username_5 on 2018-02-27. Score: 2
  
  If you are filling up your docker entrypoint with `sed` commands, consider this practice:
  
  * Use a tool such as [confd](https://github.com/kelseyhightower/confd) to manage your docker images configuration files and keep them updated
  
  [Confd](https://github.com/kelseyhightower/confd) will read data from many supported key-value stores and render configuration templates dynamically.
  
  
  ---
  
  ### Answer by username_1 on 2019-05-13. Score: 0
  
  One could use [A2D](https://github.com/030/a2d) to bake an app into a docker image while taking certain things into account, e.g. non-root, permissions, location of the app:
  
  > ```
  > docker run -v $PWD:/projectName utrecht/a2d:1.0.0 \
  >        -projectName someProjectName -dockerfile /projectName/Dockerfile
  >
  > ```
  
  returns:
  
  > ```
  > FROM golang:1.12.4-alpine as builder
  > COPY . ./someProjectName/
  > WORKDIR someProjectName
  > RUN adduser -D -g '' someProjectName && \
  >     apk add git && \
  >     CGO_ENABLED=0 go build && \
  >     cp someProjectName /someProjectName && \
  >     chmod 100 /someProjectName
  >
  > FROM scratch
  > COPY --from=builder /etc/group /etc/group
  > COPY --from=builder /etc/passwd /etc/passwd
  > COPY --from=builder --chown=someProjectName:someProjectName /someProjectName /usr/local/someProjectName
  > COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
  > USER someProjectName
  > ENTRYPOINT ["/usr/local/someProjectName"]
  >
  > ```
  
  
  ---
  
  ### Answer by username_1 on 2017-09-07. Score: 0
  
  Reading [the official docker security article](https://docs.docker.com/engine/security/security/) and comparing it with the bullets that are defined in the question the following things are important as well:
  
  * protect the docker daemon socket by implementing ssl
  * content trust using notary and `DOCKER_CONTENT_TRUST` variable
  
  > **Comments:**
  > **username_7**: You should try to compile your answer, this part could be within any of your other two answers  
  
  ---
  
  