Answers: <p>The docker cache needs the following:</p>
  
  <ul>
  <li>Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and</li>
  <li>The previous image needs to still be on the build host (not pruned) and</li>
  <li>The same previous layer and either</li>
  <li>The same command being run (including the same environment/args) or</li>
  <li>The same hash on the files being copied</li>
  </ul>
  
  <p>If you see the cache being busted on a <code>COPY</code> or <code>ADD</code> command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a <code>docker image history</code> command, e.g.:</p>
  
  <pre><code>$ docker image history golang:alpine
  IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
  6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
  &lt;missing&gt;           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] &amp;&amp; ec…   17B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
  &lt;missing&gt;           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
  &lt;missing&gt;           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB
  </code></pre>
  
  <p>You can include the <code>--no-trunc</code> option to get a full line. The key piece of data relevant to your question from the above image is in the <code>ADD</code> step where you see <code>file:0eb5ea35741d23fe3…</code>. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-03-22T16:18:26.940Z
Answers: Could you be more specific on this statement, please? "or you explicitly added flags to trust a pulled image" For Gitlab CI pipeline development I build an image locally and push it into the Gitlab registry. The remote CI does not pick up the cache, while the layers are cached for local builds.
Answers: Stefan_Fairphone
Answers: CC BY-SA 4.0
Answers: 2022-03-22T16:24:14.493Z
Answers: @Stefan_Fairphone there's a `--cache-from` flag that marks an image you pulled as being trusted for reusing the layers.
Answers: BMitch
Answers: CC BY-SA 4.0
Answers: 2022-03-23T08:16:15.127Z
Answers: Thank you for your prompt confirmation. It seems I might be affected by a BuildKit issue: [Inconsistent cache behavior from CI and local environment (or from env to env) · Issue #2010 · moby/buildkit](https://github.com/moby/buildkit/issues/2010)
Answers: Stefan_Fairphone
Answers: CC BY-SA 4.0
Answers: 2019-08-02T15:05:56.330Z
Answers: 2019-08-02T15:05:56.330Z
Answers: BMitch
Answers: <p>Did you try</p>
  <pre><code>docker inspect TAG
  </code></pre>
  <p>Also try the manual binary search method in the answer by Timmay to Server Fault StackExchange question <a href="https://serverfault.com/questions/747807/how-to-debug-docker-cache-invalidation">&quot;How to debug Docker cache invalidation?&quot;</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2021-02-18T15:20:20.697Z
Answers: 2021-02-18T15:20:20.697Z
Answers: HenryM
Answers: <p>I have to deal with that same issue and the main problem is not the cache itself is the time that takes to make a build but also the problem of being sure you have the right changes on the image you are building.</p>
  
  <p>I can recommend you some different options you can make separately or all together:</p>
  
  <ul>
  <li><p>Make cronjob that cleans your docker Daemon of old images periodically, for example, once a week. With that, you are sure to have fresh images every week and have a good performance with the cache for the daily build, the only one that takes a little bit more time is the first build after the flush.</p></li>
  <li><p>Make your own base image with all the content you know you don't change too often, for that you can have more than one Dockerfile on your project and launch a build process of the base image after the flush on the previous step to be sure you have the latest changes on a fresh image</p></li>
  <li><p>Use multi-stage build or build patter to have a good size image at the end of the build.</p></li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2019-08-02T07:16:03.030Z
Answers: What you recommend is exactly what we do. The problem is in my last two sentences - this rarely changed stuff DOESN'T change, but the cache is still invalidated, which is what I am attempting to debug.
Answers: Torque
Answers: CC BY-SA 4.0
Answers: 2019-08-01T20:35:56.073Z
Answers: 2019-08-01T21:17:20.067Z
Answers: 2019-08-01T21:17:20.067Z
Answers: wolmi
Answers: <p>According to Docker documentation <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/" rel="nofollow noreferrer">search for: "Leverage build cache"</a>, there is checksum calculated on the directory - please see the article for details.</p>
  
  <p>What I will suggest is a multistage build.</p>
  
  <p>If we have static content of directory we could create container/image with its content. Then instead of using Docker copy, we could use FROM as a build stage and copy that directory into a new layer, which in fact will reuse that layer.
  That is based on a fact that we don't operate on a directory (when a file could get a changed attribute like last accessed time). </p>
  
  <h2>Base Dockerfile</h2>
  
  <pre><code>FROM alpine:latest  
  COPY /myDirectory .
  </code></pre>
  
  <p>Then you could tag this container as storage-base
  O</p>
  
  <h2>MultiStage Dockerfile</h2>
  
  <pre><code>FROM storage-base AS storage
  
  FROM alpine:latest  
  RUN a command 
  WORKDIR /root/
  COPY --from=storage /myDirectory .
  </code></pre>
  
  <p>Please check more references here: <a href="https://docs.docker.com/develop/develop-images/multistage-build/" rel="nofollow noreferrer">Docker Multistage Build</a></p>
  
  <h2>Edit</h2>
  
  <ol>
  <li>the image/container created in step 1 shall be only recreated when
  directory content will change (by a trigger that knows that the data changed), so that means it will not be created on every build.</li>
  <li>then we use the image that was created once (and re-use that layer)</li>
  </ol>
  
  <p>I just created a simple example to help express my idea - <a href="https://github.com/profesor79/docker-cache-directory-as-an-image-multistage-build" rel="nofollow noreferrer">Git repo</a> - run build-docker.sh script</p>
  
  <p>I found that on reboot Docker is re-creating cache instead of reusing, but it takes just a few minutes and it is not critical.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-08-01T14:07:48.800Z
Answers: Unfortunately, this doesn't help me at all. I did look at the article in my own research. It gives no detail and no help in debugging why a directory that is bitwise identical does not get cached. Splitting the build up into multiple parts also would not help, since the first stage would get the cache busted triggering all later stages being uncached as well :(
Answers: Torque
Answers: CC BY-SA 4.0
Answers: 2019-08-01T17:03:07.960Z
Answers: @Torque please see my edit, as I poorly expressed my thoughts.
Answers: profesor79
Answers: CC BY-SA 4.0
Answers: 2019-08-02T07:14:37.497Z
Answers: While I appreciate the effort you took, this really doesn't address the issue - writing your own programs or scripts to get around a bug in Dockers caching is not a sane way. Docker already does have this functionality, and I'm looking for ways to debug why it's not working as defined, not hack my way around it. Furthermore, like I said, we are using this in CI/CD so no old files would be there, it's always a fresh checkout. Lastly, a cache invalidation adds over an hour of compile time to our builds, not a few minutes.
Answers: Torque
Answers: CC BY-SA 4.0
Answers: 2019-08-02T08:05:03.800Z
Answers: @Torque thanks for the comment. Is your build system restarted?
Answers: profesor79
Answers: CC BY-SA 4.0
Answers: 2019-08-02T09:19:14.903Z
Answers: Restarts are not what invalidate the cache, if that is what you are asking.
Answers: Torque
Answers: CC BY-SA 4.0
Answers: 2019-08-01T13:03:59.127Z
Answers: 2019-08-01T17:02:28.277Z
Answers: 2019-08-01T17:02:28.277Z
Answers: profesor79
Answers: <p>As I've discussed in <a href="https://www.augmentedmind.de/2023/01/08/docker-build-cache-debug/" rel="nofollow noreferrer">this blog post</a>, I found the following reasons for (unexpected) cache misses in Docker:</p>
  <ul>
  <li>If the problem happens in a <em>CI pipeline</em>, and if you use <em>multiple</em> build agent machines (or &quot;ephemeral&quot; machines), it might happen that build job #1 was executed on agent #A, but build job #2 was executed on agent #B, which has a different local cache than agent #A. Consequently, when you look at your CI pipeline output, be sure to check on which agent the jobs are executed. To prevent these kinds of cache misses, you can use a remote cache, storing caching information in a remote image registry. There are two implementation approaches: <em>inline</em> caching (where the image builder embeds caching meta-data into the image it builds), or using a separate <em>registry cache</em> (where a separate image is pushed that contains only cache blobs). The usage details of remote caching depend on your image builder tool. For instance, <code>docker build</code> supports only inline caching (see <a href="https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources" rel="nofollow noreferrer">here</a>), <code>docker buildx</code> (or when using BuildKit directly) supports both approaches (see <a href="https://docs.docker.com/build/cache/backends/" rel="nofollow noreferrer">here</a>), Buildah and kaniko only support the registry cache.</li>
  <li>If you use <code>ARG</code> in your Dockerfile, it is easy to accidentally break the cache invalidation. Whenever the value of some <code>ARG</code> is different between two <code>docker build</code> executions, the second execution won’t be able to reuse the previously cached layer for a <code>RUN</code> or <code>ENV</code> command that <strong>uses</strong> the <code>ARG</code>'s value. This then also invalidates all follow-up layers. See <a href="https://docs.docker.com/engine/reference/builder/#impact-on-build-caching" rel="nofollow noreferrer">here</a> for background information. If you use <em>multi-stage builds</em>, and if you run <code>docker build</code> <em>several</em> times (for different <em>targets</em>), make sure you <strong>always</strong> provide the same <em>ARG</em> values to <em>all</em> docker build calls!</li>
  <li>Sometimes the entire image is rebuilt whenever a new <strong>base</strong> image has been released (that you reference in a <code>FROM</code> statement). This particularly happens if you use <code>docker build --pull</code>. You need to closely look at the builder’s output of the <em>first</em> layer, which includes the SHA-256 checksum of the base image. If it keeps changing frequently, there is no real &quot;fix&quot;. Your image <em>should</em> be rebuilt, to include the most recent security fixes of the base image. However, if the base image is rebuilt <em>very often</em> (e.g. multiple times per day), you may want to stop using the <code>--pull</code> flag, and instead have a different approach that only runs <code>docker pull &lt;base image&gt;</code> (or <em>delete</em> the base image) more rarely, e.g. once per day.</li>
  <li>Layers for <code>COPY</code> or <code>ADD</code> statements are rebuilt &quot;unexpectedly&quot; whenever files change that you did not have on your radar (which you did not include into your <code>.dockerignore</code> yet). This could be the <code>.git</code> folder, or files created during building/testing (e.g. unit test report files, or log files). This typically happens when running <code>COPY . .</code>, because then your entire project directory is copied from the build context into the build container, which increases the chance that you missed excluding some superfluous files (that do not belong into the container anyway) via <code>.dockerignore</code>. To make it easy to address this issue, I developed a small CLI tool called <a href="https://github.com/MShekow/directory-checksum/" rel="nofollow noreferrer">Directory Checksum</a>. It recursively computes the checksum of the contents of a directory, and prints the checksums up to a depth you can specify.</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2023-01-08T10:30:41.893Z
Answers: 2023-01-08T10:30:41.893Z
Answers: MShekow
Body: <p>I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.</p>
  
Comments: CC BY-SA 4.0
Comments: 2021-01-01T06:31:14.783Z
Comments: Debugging tip- you can check what's equivalent (contents, dates, owners, and perms) or not by using `tar -cf - <directory or file> | md5sum`. I built twice, then I started at the top level directories in my jenkins workspaces and worked my way down until I found where hashes started matching, then I worked up until I found what didn't. In my case, I had to set a couple git directories' last modified time.
Comments: bkribbs
Tags: <docker>
Title: Debug Docker caching issues
UserDisplayName: Torque
ThreadText: # Debug Docker caching issues
  
  (Asked by: username_0 on 2019-08-01)
  
  I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.
  
  **Tags:** docker
  
  > **Comments:**
  > **username_1**: Debugging tip- you can check what's equivalent (contents, dates, owners, and perms) or not by using `tar -cf - <directory or file> | md5sum`. I built twice, then I started at the top level directories in my jenkins workspaces and worked my way down until I found where hashes started matching, then I worked up until I found what didn't. In my case, I had to set a couple git directories' last modified time.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_2 on 2019-08-02. Score: 3
  
  The docker cache needs the following:
  
  * Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and
  * The previous image needs to still be on the build host (not pruned) and
  * The same previous layer and either
  * The same command being run (including the same environment/args) or
  * The same hash on the files being copied
  
  If you see the cache being busted on a `COPY` or `ADD` command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a `docker image history` command, e.g.:
  
  ```
  $ docker image history golang:alpine
  IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
  6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
  <missing>           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
  <missing>           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
  <missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
  <missing>           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
  <missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
  <missing>           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
  <missing>           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
  <missing>           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
  <missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB
  
  ```
  
  You can include the `--no-trunc` option to get a full line. The key piece of data relevant to your question from the above image is in the `ADD` step where you see `file:0eb5ea35741d23fe3…`. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.
  
  > **Comments:**
  > **username_3**: Could you be more specific on this statement, please? "or you explicitly added flags to trust a pulled image" For Gitlab CI pipeline development I build an image locally and push it into the Gitlab registry. The remote CI does not pick up the cache, while the layers are cached for local builds.  
  > **username_2**: @username_3 there's a `--cache-from` flag that marks an image you pulled as being trusted for reusing the layers.  
  > **username_3**: Thank you for your prompt confirmation. It seems I might be affected by a BuildKit issue: [Inconsistent cache behavior from CI and local environment (or from env to env) · Issue #2010 · moby/buildkit](https://github.com/moby/buildkit/issues/2010)  
  
  ---
  
  ### Answer by username_4 on 2021-02-18. Score: 0
  
  Did you try
  
  ```
  docker inspect TAG
  
  ```
  
  Also try the manual binary search method in the answer by Timmay to Server Fault StackExchange question ["How to debug Docker cache invalidation?"](https://serverfault.com/questions/747807/how-to-debug-docker-cache-invalidation)
  
  
  ---
  
  ### Answer by username_5 on 2019-08-01. Score: 0
  
  I have to deal with that same issue and the main problem is not the cache itself is the time that takes to make a build but also the problem of being sure you have the right changes on the image you are building.
  
  I can recommend you some different options you can make separately or all together:
  
  * Make cronjob that cleans your docker Daemon of old images periodically, for example, once a week. With that, you are sure to have fresh images every week and have a good performance with the cache for the daily build, the only one that takes a little bit more time is the first build after the flush.
  * Make your own base image with all the content you know you don't change too often, for that you can have more than one Dockerfile on your project and launch a build process of the base image after the flush on the previous step to be sure you have the latest changes on a fresh image
  * Use multi-stage build or build patter to have a good size image at the end of the build.
  
  > **Comments:**
  > **username_0**: What you recommend is exactly what we do. The problem is in my last two sentences - this rarely changed stuff DOESN'T change, but the cache is still invalidated, which is what I am attempting to debug.  
  
  ---
  
  ### Answer by username_6 on 2019-08-01. Score: 0
  
  According to Docker documentation [search for: "Leverage build cache"](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/), there is checksum calculated on the directory - please see the article for details.
  
  What I will suggest is a multistage build.
  
  If we have static content of directory we could create container/image with its content. Then instead of using Docker copy, we could use FROM as a build stage and copy that directory into a new layer, which in fact will reuse that layer.
  That is based on a fact that we don't operate on a directory (when a file could get a changed attribute like last accessed time).
  
  ## Base Dockerfile
  
  ```
  FROM alpine:latest  
  COPY /myDirectory .
  
  ```
  
  Then you could tag this container as storage-base
  O
  
  ## MultiStage Dockerfile
  
  ```
  FROM storage-base AS storage
  
  FROM alpine:latest  
  RUN a command 
  WORKDIR /root/
  COPY --from=storage /myDirectory .
  
  ```
  
  Please check more references here: [Docker Multistage Build](https://docs.docker.com/develop/develop-images/multistage-build/)
  
  ## Edit
  
  1. the image/container created in step 1 shall be only recreated when
     directory content will change (by a trigger that knows that the data changed), so that means it will not be created on every build.
  2. then we use the image that was created once (and re-use that layer)
  
  I just created a simple example to help express my idea - [Git repo](https://github.com/username_6/docker-cache-directory-as-an-image-multistage-build) - run build-docker.sh script
  
  I found that on reboot Docker is re-creating cache instead of reusing, but it takes just a few minutes and it is not critical.
  
  > **Comments:**
  > **username_0**: Unfortunately, this doesn't help me at all. I did look at the article in my own research. It gives no detail and no help in debugging why a directory that is bitwise identical does not get cached. Splitting the build up into multiple parts also would not help, since the first stage would get the cache busted triggering all later stages being uncached as well :(  
  > **username_6**: @username_0 please see my edit, as I poorly expressed my thoughts.  
  > **username_0**: While I appreciate the effort you took, this really doesn't address the issue - writing your own programs or scripts to get around a bug in Dockers caching is not a sane way. Docker already does have this functionality, and I'm looking for ways to debug why it's not working as defined, not hack my way around it. Furthermore, like I said, we are using this in CI/CD so no old files would be there, it's always a fresh checkout. Lastly, a cache invalidation adds over an hour of compile time to our builds, not a few minutes.  
  > **username_6**: @username_0 thanks for the comment. Is your build system restarted?  
  > **username_0**: Restarts are not what invalidate the cache, if that is what you are asking.  
  
  ---
  
  ### Answer by username_7 on 2023-01-08. Score: 0
  
  As I've discussed in [this blog post](https://www.augmentedmind.de/2023/01/08/docker-build-cache-debug/), I found the following reasons for (unexpected) cache misses in Docker:
  
  * If the problem happens in a *CI pipeline*, and if you use *multiple* build agent machines (or "ephemeral" machines), it might happen that build job #1 was executed on agent #A, but build job #2 was executed on agent #B, which has a different local cache than agent #A. Consequently, when you look at your CI pipeline output, be sure to check on which agent the jobs are executed. To prevent these kinds of cache misses, you can use a remote cache, storing caching information in a remote image registry. There are two implementation approaches: *inline* caching (where the image builder embeds caching meta-data into the image it builds), or using a separate *registry cache* (where a separate image is pushed that contains only cache blobs). The usage details of remote caching depend on your image builder tool. For instance, `docker build` supports only inline caching (see [here](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources)), `docker buildx` (or when using BuildKit directly) supports both approaches (see [here](https://docs.docker.com/build/cache/backends/)), Buildah and kaniko only support the registry cache.
  * If you use `ARG` in your Dockerfile, it is easy to accidentally break the cache invalidation. Whenever the value of some `ARG` is different between two `docker build` executions, the second execution won’t be able to reuse the previously cached layer for a `RUN` or `ENV` command that **uses** the `ARG`'s value. This then also invalidates all follow-up layers. See [here](https://docs.docker.com/engine/reference/builder/#impact-on-build-caching) for background information. If you use *multi-stage builds*, and if you run `docker build` *several* times (for different *targets*), make sure you **always** provide the same *ARG* values to *all* docker build calls!
  * Sometimes the entire image is rebuilt whenever a new **base** image has been released (that you reference in a `FROM` statement). This particularly happens if you use `docker build --pull`. You need to closely look at the builder’s output of the *first* layer, which includes the SHA-256 checksum of the base image. If it keeps changing frequently, there is no real "fix". Your image *should* be rebuilt, to include the most recent security fixes of the base image. However, if the base image is rebuilt *very often* (e.g. multiple times per day), you may want to stop using the `--pull` flag, and instead have a different approach that only runs `docker pull <base image>` (or *delete* the base image) more rarely, e.g. once per day.
  * Layers for `COPY` or `ADD` statements are rebuilt "unexpectedly" whenever files change that you did not have on your radar (which you did not include into your `.dockerignore` yet). This could be the `.git` folder, or files created during building/testing (e.g. unit test report files, or log files). This typically happens when running `COPY . .`, because then your entire project directory is copied from the build context into the build container, which increases the chance that you missed excluding some superfluous files (that do not belong into the container anyway) via `.dockerignore`. To make it easy to address this issue, I developed a small CLI tool called [Directory Checksum](https://github.com/username_7/directory-checksum/). It recursively computes the checksum of the contents of a directory, and prints the checksums up to a depth you can specify.
  
  
  ---
  
  