Answers: <p>Perhaps you are looking for the <a href="https://plugins.jenkins.io/jobcacher" rel="nofollow noreferrer">Job Cacher</a> plugin?</p>
  <p>To quote the plugin page:</p>
  <blockquote>
  <p><strong>Summary</strong></p>
  <p>Enables caching of files on transient executors for use on subsequent builds to improve performance</p>
  <p><strong>Features</strong></p>
  <ul>
  <li>Item storage extension point supporting on master storage and AWS S3</li>
  <li>Cache Wrapper for free style jobs that manages the cache
  Arbitrary File Cache where user specifies paths to be cached</li>
  <li>UI on Job page to review the job's caches</li>
  <li>Supports Pipeline jobs with a cache directive</li>
  <li>Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches</li>
  <li>Incremental storing of cache into storage to minimize data movement</li>
  </ul>
  </blockquote>
  <p>However, these are for files <strong>created</strong> by the jobs, not <strong>needed</strong> by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.</p>
  <p>To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the <em>general</em> problem is <strong>how to cache job dependencies</strong> in Jenkins. If you agree to this wider interpretation, there are some other options.</p>
  <p>I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins.  If you need really <em>arbitrary</em> dependency management, a tool like <a href="https://spack.readthedocs.io" rel="nofollow noreferrer">Spack</a> or <a href="https://easybuild.readthedocs.io/en/latest/" rel="nofollow noreferrer">EasyBuild</a> may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.</p>
  <p>However, since you mention just a few dependencies (specific sqlite versions <em>e.g.</em>), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-11-05T05:27:45.183Z
Answers: That seems like a means to cache _my own_ artifact -- file produced by my build. What I'm looking for is a way to efficiently download somebody else's tarball -- from afar... I know, I can write a script of my own, but I'm puzzled, why -- given their wide variety -- there is no plugin for this among the existing ones. FreeBSD's ports-collection's been doing just this (downloading, verifying, and building 3rd-party sources) for decades, for example...
Answers: Mikhail T.
Answers: CC BY-SA 4.0
Answers: 2019-11-04T06:46:31.560Z
Answers: 2019-11-04T06:46:31.560Z
Answers: 2020-06-18T08:28:10.803Z
Answers: Bruce Becker
Body: <p>Although <em>our own</em> sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.</p>
  
  <p>The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:</p>
  
  <ul>
  <li><a href="https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
  <li><a href="http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
  <li><a href="http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
  </ul>
  
  <p>and its digest (such as SHA256).</p>
  
  <p>At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.</p>
  
  <p>I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.</p>
  
  <p>Any help?</p>
  
Tags: <jenkins><jenkins-plugins>
Title: Is there a Jenkins plugin for downloading source code?
UserDisplayName: Mikhail T.
ThreadText: # Is there a Jenkins plugin for downloading source code?
  
  (Asked by: username_0 on 2019-11-03)
  
  Although *our own* sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.
  
  The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:
  
  * <https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>
  * <http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>
  * <http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>
  
  and its digest (such as SHA256).
  
  At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.
  
  I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.
  
  Any help?
  
  **Tags:** jenkins, jenkins-plugins
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-04. Score: 1
  
  Perhaps you are looking for the [Job Cacher](https://plugins.jenkins.io/jobcacher) plugin?
  
  To quote the plugin page:
  
  > **Summary**
  >
  > Enables caching of files on transient executors for use on subsequent builds to improve performance
  >
  > **Features**
  >
  > * Item storage extension point supporting on master storage and AWS S3
  > * Cache Wrapper for free style jobs that manages the cache
  >   Arbitrary File Cache where user specifies paths to be cached
  > * UI on Job page to review the job's caches
  > * Supports Pipeline jobs with a cache directive
  > * Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
  > * Incremental storing of cache into storage to minimize data movement
  
  However, these are for files **created** by the jobs, not **needed** by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.
  
  To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the *general* problem is **how to cache job dependencies** in Jenkins. If you agree to this wider interpretation, there are some other options.
  
  I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins. If you need really *arbitrary* dependency management, a tool like [Spack](https://spack.readthedocs.io) or [EasyBuild](https://easybuild.readthedocs.io/en/latest/) may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.
  
  However, since you mention just a few dependencies (specific sqlite versions *e.g.*), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.
  
  > **Comments:**
  > **username_0**: That seems like a means to cache _my own_ artifact -- file produced by my build. What I'm looking for is a way to efficiently download somebody else's tarball -- from afar... I know, I can write a script of my own, but I'm puzzled, why -- given their wide variety -- there is no plugin for this among the existing ones. FreeBSD's ports-collection's been doing just this (downloading, verifying, and building 3rd-party sources) for decades, for example...  
  
  ---
  
  