Answers: <blockquote>
  <p>Does this procedure make sense as a solution to my use case?</p>
  </blockquote>
  <p>vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.</p>
  <blockquote>
  <p>I can't imagine being the first to realize this use case. Is there an
  existing mechanism (as part of the docker utilities, as part of the
  Azure DevOps Pipelines framework or from something completely
  different) that fulfills my needs?</p>
  </blockquote>
  <p>I'm currently doing this in some of my <a href="https://github.com/DontShaveTheYak/terraform-module-template/blob/main/.github/workflows/test.yml" rel="nofollow noreferrer">side projects</a>. It is also supported on <a href="https://github.com/devcontainers/ci/blob/main/docs/azure-devops-task.md" rel="nofollow noreferrer">azure</a>.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-02T06:58:42.693Z
Answers: As mentioned above my questions are about using the `Dockerfile` hash to tag the docker image to get a commit-specific build environment without building the docker image on each and every commit.
Answers: Roland Sarrazin
Answers: CC BY-SA 4.0
Answers: 2022-08-02T05:25:56.303Z
Answers: 2022-08-02T05:25:56.303Z
Answers: Levi
Answers: <p>Docker has this capability built-in and you can use it in a variety of ways.</p>
  <p>One way would be to keep a &quot;cache&quot; tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.</p>
  <pre><code>docker pull ${MY_IMAGE}:cache || true
  docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
  docker push ${MY_IMAGE}:${VERSION}
  docker push ${MY_IMAGE}:cache
  </code></pre>
  <p>If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.</p>
  <pre><code>docker pull ${MY_IMAGE}:latest || true
  docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
  docker push ${MY_IMAGE}:${VERSION}
  </code></pre>
  <p>This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.</p>
  <p>Then the <code>--cache-from</code> flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-04T09:25:10.677Z
Answers: You would then put the `Dockerfile` hash into `$VERSION`, correct? I fail at understanding how old builds would be able to use the old image (i.e. the very same image that used to be the current one at that time) and not the current latest one.
Answers: Roland Sarrazin
Answers: CC BY-SA 4.0
Answers: 2022-08-04T18:14:11.643Z
Answers: I guess it's a bad example because in my pipeline I'm building and deploying an application using semantic versioning. 
  
  But it doesn't matter what you label the new image as, you can even remove the version label. 
  
  The magic is in --cache-from. Cache from makes sure that docker doesn't build image layers that didnt chage.
Answers: Levi
Answers: CC BY-SA 4.0
Answers: 2022-08-05T09:42:31.540Z
Answers: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources seems to require `--build-arg BUILDKIT_INLINE_CACHE=1` to be set when building the initial image.
Answers: Roland Sarrazin
Answers: CC BY-SA 4.0
Answers: 2022-08-05T14:51:22.653Z
Answers: Why dont you just try it @RolandSarrazin ? This command works for me and I am not using build args and passing BUILDKIT_INLINE_CACHE in my image. I think that is required for external cache sources and not for local caches.
Answers: Levi
Answers: CC BY-SA 4.0
Answers: 2022-08-02T13:12:11.770Z
Answers: 2022-08-02T13:12:11.770Z
Answers: Levi
Body: <p>Given a repository containing a <code>Dockerfile</code> that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a <a href="https://code.visualstudio.com/docs/remote/create-dev-container" rel="nofollow noreferrer">Visual Studio Code devcontainer</a>), the CI pipeline shall fulfill the following requirements:</p>
  <ol>
  <li>The CI pipeline shall build the repository's contents in the context of a docker image built from the very <code>Dockerfile</code> present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).</li>
  </ol>
  <ul>
  <li>Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.</li>
  <li>Rationale: Changes to the <code>Dockerfile</code> shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).</li>
  </ul>
  <ol start="2">
  <li>The CI pipeline shall reuse an already built docker image if the applicable <code>Dockerfile</code> has already been &quot;baked&quot; into such an image pushed to the registry.</li>
  </ol>
  <ul>
  <li>Rationale: Building a docker image on each commit is resource-consuming (time, energy).</li>
  </ul>
  <p>I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:</p>
  <ul>
  <li>Calculate the <code>Dockerfile</code>'s hash.</li>
  <li>Load <code>docker_image_name:$hash</code> from the docker registry.</li>
  <li>If the load fails, build <code>docker_image_name:$hash</code> from the <code>Dockerfile</code> and push it to the docker registry.</li>
  <li>Use <code>docker_image_name:$hash</code> (from the registry / from the local cache) to run the CI pipeline (using Azure's <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops#endpoints" rel="nofollow noreferrer">container jobs</a> in my case).</li>
  </ul>
  <p>Questions:</p>
  <ol>
  <li>Does this procedure make sense as a solution to my use case?</li>
  <li>I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?</li>
  </ol>
  
Comments: CC BY-SA 4.0
Comments: 2022-08-02T05:28:46.440Z
Comments: I wasn't exactly sure if you was asking how to use a vscode dev container in a azure pipeline or if you was asking how to replicate that dev container functionality from scratch.
Comments: Levi
Comments: CC BY-SA 4.0
Comments: 2022-08-02T06:53:09.913Z
Comments: @Levi: My mentioning vscode's devcontainers might have been misleading since it was just part of the context, but not part of the actual question. The question was specifically about how to use the very docker image specified by the currently valid `Dockerfile` without building the image on each and every commit.
Comments: Roland Sarrazin
Comments: CC BY-SA 4.0
Comments: 2022-08-02T13:13:43.387Z
Comments: I think the question boils down to "How do I avoid rebuilding my container image in CICD" which probably has been asked and answered before. I went ahead and gave a new answer though.
Comments: Levi
Tags: <docker><continuous-integration><dockerfile><azure-pipelines>
Title: How do I run a CI build in a docker image matching the current `Dockerfile` while being resource-aware?
UserDisplayName: Roland Sarrazin
ThreadText: # How do I run a CI build in a docker image matching the current `Dockerfile` while being resource-aware?
  
  (Asked by: username_0 on 2022-08-01)
  
  Given a repository containing a `Dockerfile` that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a [Visual Studio Code devcontainer](https://code.visualstudio.com/docs/remote/create-dev-container)), the CI pipeline shall fulfill the following requirements:
  
  1. The CI pipeline shall build the repository's contents in the context of a docker image built from the very `Dockerfile` present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).
  
  * Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.
  * Rationale: Changes to the `Dockerfile` shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).
  
  2. The CI pipeline shall reuse an already built docker image if the applicable `Dockerfile` has already been "baked" into such an image pushed to the registry.
  
  * Rationale: Building a docker image on each commit is resource-consuming (time, energy).
  
  I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:
  
  * Calculate the `Dockerfile`'s hash.
  * Load `docker_image_name:$hash` from the docker registry.
  * If the load fails, build `docker_image_name:$hash` from the `Dockerfile` and push it to the docker registry.
  * Use `docker_image_name:$hash` (from the registry / from the local cache) to run the CI pipeline (using Azure's [container jobs](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops#endpoints) in my case).
  
  Questions:
  
  1. Does this procedure make sense as a solution to my use case?
  2. I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?
  
  **Tags:** docker, continuous-integration, dockerfile, azure-pipelines
  
  > **Comments:**
  > **username_1**: I wasn't exactly sure if you was asking how to use a vscode dev container in a azure pipeline or if you was asking how to replicate that dev container functionality from scratch.  
  > **username_0**: @username_1: My mentioning vscode's devcontainers might have been misleading since it was just part of the context, but not part of the actual question. The question was specifically about how to use the very docker image specified by the currently valid `Dockerfile` without building the image on each and every commit.  
  > **username_1**: I think the question boils down to "How do I avoid rebuilding my container image in CICD" which probably has been asked and answered before. I went ahead and gave a new answer though.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-08-02. Score: 0
  
  > Does this procedure make sense as a solution to my use case?
  
  vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.
  
  > I can't imagine being the first to realize this use case. Is there an
  > existing mechanism (as part of the docker utilities, as part of the
  > Azure DevOps Pipelines framework or from something completely
  > different) that fulfills my needs?
  
  I'm currently doing this in some of my [side projects](https://github.com/DontShaveTheYak/terraform-module-template/blob/main/.github/workflows/test.yml). It is also supported on [azure](https://github.com/devcontainers/ci/blob/main/docs/azure-devops-task.md).
  
  > **Comments:**
  > **username_0**: As mentioned above my questions are about using the `Dockerfile` hash to tag the docker image to get a commit-specific build environment without building the docker image on each and every commit.  
  
  ---
  
  ### Answer by username_1 on 2022-08-02. Score: 0
  
  Docker has this capability built-in and you can use it in a variety of ways.
  
  One way would be to keep a "cache" tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.
  
  ```
  docker pull ${MY_IMAGE}:cache || true
  docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
  docker push ${MY_IMAGE}:${VERSION}
  docker push ${MY_IMAGE}:cache
  
  ```
  
  If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.
  
  ```
  docker pull ${MY_IMAGE}:latest || true
  docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
  docker push ${MY_IMAGE}:${VERSION}
  
  ```
  
  This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.
  
  Then the `--cache-from` flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.
  
  > **Comments:**
  > **username_0**: You would then put the `Dockerfile` hash into `$VERSION`, correct? I fail at understanding how old builds would be able to use the old image (i.e. the very same image that used to be the current one at that time) and not the current latest one.  
  > **username_1**: I guess it's a bad example because in my pipeline I'm building and deploying an application using semantic versioning.   But it doesn't matter what you label the new image as, you can even remove the version label.   The magic is in --cache-from. Cache from makes sure that docker doesn't build image layers that didnt chage.  
  > **username_0**: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources seems to require `--build-arg BUILDKIT_INLINE_CACHE=1` to be set when building the initial image.  
  > **username_1**: Why dont you just try it @username_0 ? This command works for me and I am not using build args and passing BUILDKIT_INLINE_CACHE in my image. I think that is required for external cache sources and not for local caches.  
  
  ---
  
  