Answers: <p>It looks like what you're looking for is less about caching and more about mirroring. Because your requirements have to deal with permanently unavailable packages, you need to persist the packages yourself in something more permanent than a cache. In the past, we've stored packages in persistent, cloud-based stores like S3. However, there is nothing to preclude you from setting up a custom filestore on the build server.</p>
  
  <p>More concretely, it's probably easiest to set up a proxy service that looks for a resource in your filestore (S3) and returns the package if it's found. Otherwise, it would just request the resource from the upstream, populate your filestore, and return the package.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-04-18T20:34:01.483Z
Answers: 2017-04-18T20:34:01.483Z
Answers: Blue Matador
Answers: <p>I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).</p>
  
  <p>To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's <a href="https://www.sonatype.com/nexus-repository-software-component-management" rel="nofollow noreferrer">Nexus</a> and JFrog's <a href="https://www.jfrog.com/artifactory/" rel="nofollow noreferrer">Artifactory</a>.  Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot.  There are other options out there, I just listed these two because I've worked with them already and know they will do what you need.  As a bonus you might also be able to use them to manage the docker image directly as well.</p>
  
  <p>There are others too worth mentioning, you can find a <a href="https://binary-repositories-comparison.github.io/" rel="nofollow noreferrer">comparison table</a> for them, you'll get a better view of their capabilities. (thanks to Karl Harnagy for the link)</p>
  
  <p>Note that some of these options might require one or another of the paid versions so YMMV.</p>
  
  <p>Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.</p>
  
  <p>If none of these do it for you you can look into other technologies or roll your own.  I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.</p>
  
  <p>Hope this helps</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-04-19T13:40:51.990Z
Answers: +1 for JFrog Artifactory. Working in a highly regulated environment, we are required not to rely on external resources in our builds, in order to have a 100% reproducible build even 5 years from now. So we cache everything in Artifactory - docker base images, npm and nuget packages, 3rd party installers, etc.
Answers: yossiz74
Answers: CC BY-SA 3.0
Answers: 2017-04-19T13:44:27.030Z
Answers: nice thing about it also is that cached artifacts can generally be made available to devs directly on their system through maven, nuget, npm etc.  something a custom caching solution would require tad bit more work to get done.
Answers: Newtopian
Answers: CC BY-SA 3.0
Answers: 2017-04-19T16:14:26.603Z
Answers: Check out [ProGet](http://inedo.com/proget) you can also find a break down of all the Universal Package mangers [here](https://binary-repositories-comparison.github.io/).
Answers: Karl Harnagy
Answers: CC BY-SA 3.0
Answers: 2017-04-19T16:20:11.077Z
Answers: thanks, last I checked ProGet was pretty much limited to the .NET universe (was a while ago) I see now they are now a capable artifactory, in appearance on par with the two I mentioned.  Thanks !
Answers: Newtopian
Answers: CC BY-SA 3.0
Answers: 2017-04-18T20:57:21.083Z
Answers: 2017-04-19T16:23:48.747Z
Answers: 2017-04-19T16:23:48.747Z
Answers: Newtopian
Body: <p>A <a href="/questions/tagged/build-pipeline" class="post-tag" title="show questions tagged &#39;build-pipeline&#39;" rel="tag">build-pipeline</a> often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:</p>
  
  <ul>
  <li><p>Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.</p></li>
  <li><p>Distribution repositories are permanently unavailable, same and worse as before.</p></li>
  <li><p>Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.</p></li>
  <li><p>Some source file or binary ad-hoc package for a third-party tool is not available anymore.</p></li>
  </ul>
  
  <p>This list could be extended <em>ad lib</em> and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the <em>build-pipeline</em> is broken.</p>
  
  <p>What the is the best way to cover the two following cases when building <em>docker</em> images:</p>
  
  <ol>
  <li>Caching files downloaded with a <em>curl</em> or <em>wget</em>.</li>
  <li>Caching Debian packages.</li>
  </ol>
  
  <p>There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap <em>curl</em> calls to store results in a local cache.</p>
  
  <p>(In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)</p>
  
Tags: <docker><build-pipeline>
Title: How to cache third party resources in a build pipeline?
UserDisplayName: Michaël Le Barbier
ThreadText: # How to cache third party resources in a build pipeline?
  
  (Asked by: username_0 on 2017-04-18)
  
  A [build-pipeline](/questions/tagged/build-pipeline "show questions tagged 'build-pipeline'") often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:
  
  * Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.
  * Distribution repositories are permanently unavailable, same and worse as before.
  * Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.
  * Some source file or binary ad-hoc package for a third-party tool is not available anymore.
  
  This list could be extended *ad lib* and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the *build-pipeline* is broken.
  
  What the is the best way to cover the two following cases when building *docker* images:
  
  1. Caching files downloaded with a *curl* or *wget*.
  2. Caching Debian packages.
  
  There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap *curl* calls to store results in a local cache.
  
  (In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)
  
  **Tags:** docker, build-pipeline
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-04-18. Score: 6 | Accepted answer
  
  I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).
  
  To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's [Nexus](https://www.sonatype.com/nexus-repository-software-component-management) and JFrog's [Artifactory](https://www.jfrog.com/artifactory/). Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot. There are other options out there, I just listed these two because I've worked with them already and know they will do what you need. As a bonus you might also be able to use them to manage the docker image directly as well.
  
  There are others too worth mentioning, you can find a [comparison table](https://binary-repositories-comparison.github.io/) for them, you'll get a better view of their capabilities. (thanks to username_4 username_4 for the link)
  
  Note that some of these options might require one or another of the paid versions so YMMV.
  
  Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.
  
  If none of these do it for you you can look into other technologies or roll your own. I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.
  
  Hope this helps
  
  > **Comments:**
  > **username_3**: +1 for JFrog Artifactory. Working in a highly regulated environment, we are required not to rely on external resources in our builds, in order to have a 100% reproducible build even 5 years from now. So we cache everything in Artifactory - docker base images, npm and nuget packages, 3rd party installers, etc.  
  > **username_2**: nice thing about it also is that cached artifacts can generally be made available to devs directly on their system through maven, nuget, npm etc.  something a custom caching solution would require tad bit more work to get done.  
  > **username_4**: Check out [ProGet](http://inedo.com/proget) you can also find a break down of all the Universal Package mangers [here](https://binary-repositories-comparison.github.io/).  
  > **username_2**: thanks, last I checked ProGet was pretty much limited to the .NET universe (was a while ago) I see now they are now a capable artifactory, in appearance on par with the two I mentioned.  Thanks !  
  
  ---
  
  ### Answer by username_1 on 2017-04-18. Score: 1
  
  It looks like what you're looking for is less about caching and more about mirroring. Because your requirements have to deal with permanently unavailable packages, you need to persist the packages yourself in something more permanent than a cache. In the past, we've stored packages in persistent, cloud-based stores like S3. However, there is nothing to preclude you from setting up a custom filestore on the build server.
  
  More concretely, it's probably easiest to set up a proxy service that looks for a resource in your filestore (S3) and returns the package if it's found. Otherwise, it would just request the resource from the upstream, populate your filestore, and return the package.
  
  
  ---
  
  