Answers: <p>Docker allows you to pull a cache from anywhere with a --cache-from command.  See dockumentation here:  <a href="https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources" rel="nofollow noreferrer">https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources</a></p>
  <p>The easiest approach is to build your images with the <code>--build-arg BUILDKIT_INLINE_CACHE=1</code> flag which stores the metadata needed to build from cache. You can store these images in two places, either a file system or a registry you already have.  If you want to keep the registry of images used for deployment separate from the ones with metadata, you could separate with tags or separate registries.</p>
  <p>If you want to store the files somewhere closer you can attach volumes directly to the nodes using pvc's or shared volumes.  See this stack overflow on more details. <a href="https://stackoverflow.com/a/52564314/8954290">https://stackoverflow.com/a/52564314/8954290</a>.</p>
  <p>Here is someone who uses the same approach.  <a href="https://medium.com/swlh/fast-docker-build-in-kubernetes-f52088854f45" rel="nofollow noreferrer">https://medium.com/swlh/fast-docker-build-in-kubernetes-f52088854f45</a></p>
  <p>NOTE:  The reason that CI providers do this is to do this reliably is hard.  You must handle race conditions of which images to store as cache and which ones to pull from in the event of multiple images being built at the same time.  CI providers use workflow engines, queus, replicas etc to handle this which is not easy to build.  This is why most people use services like circle ci, codefresh etc.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-07-02T03:33:34.143Z
Answers: 2021-07-02T03:33:34.143Z
Answers: Jeff Garrett
Body: <p>We are using Azure DevOps as our CICD platform.</p>
  <p>We have our own <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops" rel="noreferrer">self hosted Linux agents</a> which we integrated into our CICD <a href="https://azure.microsoft.com/en-in/services/kubernetes-service/" rel="noreferrer">Azure Kubernetes Cluster</a> .</p>
  <p>We use these agents to build our Docker images within the cluster, meaning that we have N agents represented by N pods.</p>
  <p>While working, we might scale up and down the amount of concurrent agents depending on different parameters.</p>
  <p>Since we have started with this approach, we have encountered a huge issue in leveraging <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache" rel="noreferrer">docker's cache</a>.</p>
  <p>We have tried to:</p>
  <ol>
  <li>Use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="noreferrer">kubernetes StatefulSets</a>. But this limits us from having the cache shared between all pods (one PV/PVC per pod)</li>
  <li>Use <a href="https://github.com/GoogleContainerTools/kaniko" rel="noreferrer">Google's Kaniko</a> - Unfortunately we have found that in many cases it is slower than just using Docker's BuildKit without cache.</li>
  <li>We tried to use Azure Files with SMB and NFS as we thought that a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="noreferrer">PV with ReadWriteMany</a> will solve the problem. Unfortunately <a href="https://github.com/Azure/AKS/issues/2360" rel="noreferrer">we have ended up with a fatal error</a>.</li>
  </ol>
  <p>Currently we are a bit confused as we assume that this is a common industry issue and therefore we might have got it wrong. We are looking to understand how to leverage docker's cache while using our current setup and without moving to another platform.</p>
  <p>How should we do that? What are the best practices for this issue?</p>
  
Comments: CC BY-SA 4.0
Comments: 2021-06-06T21:08:43.017Z
Comments: Have you looked at answers to a similar question here -> https://stackoverflow.com/questions/59266670/how-to-enable-docker-layer-caching-in-azure-devops
Comments: Faheem
Comments: CC BY-SA 4.0
Comments: 2021-06-07T11:03:37.403Z
Comments: @Faheem yes I have. Unfortunately, using --cache-from means using the network (aka pulling the image) which means that local caching isn't being used. It seems like Codefresh has a solution https://codefresh.io/docs/docs/configure-ci-cd-pipeline/pipeline-caching/#distributed-docker-layer-caching which I'm curious how to implement.
Comments: Shaked KO
Tags: <docker><kubernetes><azure-devops><continuous-integration>
Title: How would you build docker images in Kubernetes while distributing its layer caching among all pod builders at scale?
UserDisplayName: Shaked KO
ThreadText: # How would you build docker images in Kubernetes while distributing its layer caching among all pod builders at scale?
  
  (Asked by: username_0 on 2021-06-05)
  
  We are using Azure DevOps as our CICD platform.
  
  We have our own [self hosted Linux agents](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops) which we integrated into our CICD [Azure Kubernetes Cluster](https://azure.microsoft.com/en-in/services/kubernetes-service/) .
  
  We use these agents to build our Docker images within the cluster, meaning that we have N agents represented by N pods.
  
  While working, we might scale up and down the amount of concurrent agents depending on different parameters.
  
  Since we have started with this approach, we have encountered a huge issue in leveraging [docker's cache](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache).
  
  We have tried to:
  
  1. Use [kubernetes StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). But this limits us from having the cache shared between all pods (one PV/PVC per pod)
  2. Use [Google's Kaniko](https://github.com/GoogleContainerTools/kaniko) - Unfortunately we have found that in many cases it is slower than just using Docker's BuildKit without cache.
  3. We tried to use Azure Files with SMB and NFS as we thought that a [PV with ReadWriteMany](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) will solve the problem. Unfortunately [we have ended up with a fatal error](https://github.com/Azure/AKS/issues/2360).
  
  Currently we are a bit confused as we assume that this is a common industry issue and therefore we might have got it wrong. We are looking to understand how to leverage docker's cache while using our current setup and without moving to another platform.
  
  How should we do that? What are the best practices for this issue?
  
  **Tags:** docker, kubernetes, azure-devops, continuous-integration
  
  > **Comments:**
  > **username_1**: Have you looked at answers to a similar question here -> https://stackoverflow.com/questions/59266670/how-to-enable-docker-layer-caching-in-azure-devops  
  > **username_0**: @username_1 yes I have. Unfortunately, using --cache-from means using the network (aka pulling the image) which means that local caching isn't being used. It seems like Codefresh has a solution https://codefresh.io/docs/docs/configure-ci-cd-pipeline/pipeline-caching/#distributed-docker-layer-caching which I'm curious how to implement.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-02. Score: 1
  
  Docker allows you to pull a cache from anywhere with a --cache-from command. See dockumentation here: <https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources>
  
  The easiest approach is to build your images with the `--build-arg BUILDKIT_INLINE_CACHE=1` flag which stores the metadata needed to build from cache. You can store these images in two places, either a file system or a registry you already have. If you want to keep the registry of images used for deployment separate from the ones with metadata, you could separate with tags or separate registries.
  
  If you want to store the files somewhere closer you can attach volumes directly to the nodes using pvc's or shared volumes. See this stack overflow on more details. <https://stackoverflow.com/a/52564314/8954290>.
  
  Here is someone who uses the same approach. <https://medium.com/swlh/fast-docker-build-in-kubernetes-f52088854f45>
  
  NOTE: The reason that CI providers do this is to do this reliably is hard. You must handle race conditions of which images to store as cache and which ones to pull from in the event of multiple images being built at the same time. CI providers use workflow engines, queus, replicas etc to handle this which is not easy to build. This is why most people use services like circle ci, codefresh etc.
  
  
  ---
  
  