title: How do I add big HTTP files in a Dockerfile and exclude them from image layers?
body: <p>Our <a href="http://www.sonatype.com/nexus/product-overview" rel="noreferrer">Nexus server</a> provides build artifacts for our Java project including its installer. That installer is <strong>really big</strong> (>1GB). I would like to retrieve and use it in a <code>Dockerfile</code>.</p>
  
  <p>What I did so far is the following:</p>
  
  <pre><code>FROM debian:jessie
  ...
  RUN apt-get install -y curl xmllib-xpath-perl
  ENV PROJECT_VERSION x.y.z-SNAPSHOT
  ...
  RUN VERSION=`curl --silent "http://nexus:8081/service/local/artifact/maven/resolve?r=public&amp;g=my.group.id&amp;a=installer&amp;v=${PROJECT_VERSION}&amp;e=sh&amp;c=linux64" | xpath -q -s '' -e '//data/version/text()'` \
      &amp;&amp; echo Version:\'${VERSION}\' \
      &amp;&amp; curl --silent http://nexus/content/groups/public/my/group/id/installer/${PROJECT_VERSION}/installer-${VERSION}-linux64.sh \
          --create-dirs \
          --output ${INSTALL_DIR}/installer.sh \
      &amp;&amp; sh ${INSTALL_DIR}/installer.sh &lt;someArgs&gt; \
      &amp;&amp; rm ${INSTALL_DIR}/installer.sh
  ...
  </code></pre>
  
  <p>With that approach I am able to:</p>
  
  <ul>
  <li>Query Nexus to provide the latest SNAPSHOT version for the provided <code>${PROJECT_VERSION}</code> which is logged out during <code>docker build</code></li>
  <li>Use that version to download the corresponding installer binary</li>
  <li>Execute the installer binary</li>
  <li>Delete the installer binary immediately after execution to <strong>not have it stored within the created Docker image layer</strong></li>
  </ul>
  
  <p>What is missing:</p>
  
  <ul>
  <li>Whenever a new installer gets deployed to Nexus I have to build the Docker image with <code>docker build --no-cache</code>. Otherwise Docker is not able to <strong>invalidate its cache</strong> and re-run the installation step for a newer installer that was meanwhile deployed to Nexus.</li>
  </ul>
  
  <p>So I tried a different approach using the <code>ADD</code> statement as those have <strong>caching capabilities</strong> according to the documentation. But that does not work since I need to provide a parameter to the <code>ADD</code> statement that is set by a previous step querying Nexus for the correct SNAPSHOT version:</p>
  
  <pre><code>FROM debian:jessie
  ...
  RUN apt-get install -y curl xmllib-xpath-perl
  ENV PROJECT_VERSION x.y.z-SNAPSHOT
  ...
  ADD http://nexus:8081/service/local/artifact/maven/resolve?r=public&amp;g=my.group.id&amp;a=installer&amp;v=${PROJECT_VERSION}&amp;e=sh&amp;c=linux64 ${INSTALL_DIR}/version.xml
  RUN cat ${INSTALL_DIR}/version.xml | xpath -q -s '' -e '//data/version/text()' &gt; ${INSTALL_DIR}/version.txt
  
  # FIXME: Somehow do a `cat ${INSTALL_DIR}/version.txt to set the ENV ${VERSION} variable ?!
  
  ADD http://nexus/content/groups/public/my/group/id/installer/${PROJECT_VERSION}/installer-${VERSION}-linux64.sh ${INSTALL_DIR}/installer.sh
  RUN ${INSTALL_DIR}/installer.sh &lt;someArgs&gt; &amp;&amp; rm ${INSTALL_DIR}/installer.sh
  ...
  </code></pre>
  
  <p>That approach does not work because:</p>
  
  <ul>
  <li>It is not possible to set the <code>${VERSION}</code> environment variable within the <code>Dockerfile</code> to the version stored within the <code>version.txt</code> file.</li>
  <li>It is not possible to prevent having the installer stored within an image layer.</li>
  </ul>
  
  <p>But at least this would use proper caching to re-use existing image layers for old installer versions and create new ones whenever a new installer version on Nexus gets deployed.</p>
  
  <p><strong>So the question is</strong>: How do I enable proper caching, cache invalidation and exclusion of the big installer file from the Docker image layers at the same time?</p>
  
  <p><strong>EDIT</strong>: I found a way to get the caching of image layers working properly by using an other Nexus API:</p>
  
  <pre><code>FROM debian:jessie
  ...
  ENV PROJECT_VERSION x.y.z-SNAPSHOT
  ...
  ADD http://nexus:8081/service/local/artifact/maven/content?r=public&amp;g=my.group.id&amp;a=installer&amp;v=${PROJECT_VERSION}&amp;e=sh&amp;c=linux64 ${INSTALL_DIR}/installer.sh
  RUN sh ${INSTALL_DIR}/installer.sh &lt;someArgs&gt; \
      &amp;&amp; rm ${INSTALL_DIR}/installer.sh
  ...
  </code></pre>
  
  <p>But still the <strong>problem of having a very big installer file included in the image layers remains</strong> since in that code snipped the <code>ADD</code> mechanism is used.</p>
  
  <p>Any ideas about how to benefit from the caching and its correct invalidation provided by the <code>ADD</code> statement but at the same time not include the added file into the images history?</p>