Answers: <p>So I believe I came up with a somewhat re-usable solution for this problem using <a href="https://pypi.org/project/devpi/" rel="nofollow noreferrer">devpi</a>. It allows you to use the same Dockerfile for the testing and production image.</p>
  <p>Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.</p>
  <p>Steps to make this work:</p>
  <p><strong>1. Base image from ARG</strong></p>
  <p>I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:</p>
  <pre><code># Builds image reistry.myorg.com/python-app
  ARG BASE=reistry.myorg.com/python-base:latest
  FROM $BASE
  
  COPY . /app
  
  # Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
  RUN pip install /app
  </code></pre>
  <p><strong>2. Upload test versions of library</strong></p>
  <p>For each python change:</p>
  <ul>
  <li>Clone the repo</li>
  <li><code>python -m build</code> the repo</li>
  <li><code>devpi upload</code> the wheels</li>
  </ul>
  <p><strong>3. Create a layer with devpi pip conf</strong></p>
  <p>Make a pip conf that can access your devpi instance and configure to your index name:</p>
  <pre><code>[global]
  index-url = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/+simple/
  trusted-host = &lt;some ip&gt;
  disable-pip-version-check = true
  
  [search]
  index = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/
  </code></pre>
  <p>And add that to your base images pip conf:</p>
  <pre><code>FROM reistry.myorg.com/python-base:latest
  
  COPY pip.conf /etc/pip.conf
  </code></pre>
  <p><strong>4. Chain the images together</strong></p>
  <pre class="lang-bash prettyprint-override"><code>docker build -t devpi-base -f devpi.Dockerfile .
  docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
  docker tag my-app reistry.myorg.com/my-app:test1
  docker push reistry.myorg.com/my-app:test1
  </code></pre>
  <hr />
  <p>Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-04-14T15:00:47.270Z
Answers: 2021-04-14T15:10:45.443Z
Answers: 2021-04-14T15:10:45.443Z
Answers: flakes
Body: <p><strong>Background</strong></p>
  <p>In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:</p>
  <ul>
  <li><code>myorg/python-lib1</code>: produces package <code>myorg.lib1</code></li>
  <li><code>myorg/python-lib2</code>: produces package <code>myorg.lib2</code></li>
  <li><code>myorg/python-app</code>: produces a package <code>myorg.app</code>, depends on <code>myorg.lib1</code> and <code>myorg.lib2</code></li>
  </ul>
  <p>In our CI for <code>myorg/python-lib1</code> and <code>myorg/python-lib1</code> we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.</p>
  <p>In our CI for <code>myorg/python-app1</code> we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:</p>
  <pre><code># Builds image reistry.myorg.com/python-app
  FROM reistry.myorg.com/python-base:latest
  
  COPY . /app
  
  # Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
  RUN pip install /app
  </code></pre>
  <p><em><strong>Note</strong>: We also have some projects using build-packs to achieve a similar goal.</em></p>
  <hr />
  <p><strong>Issue</strong></p>
  <p>Because we only run the integration tests for <code>myorg/python-app1</code>, we can't get as much confidence when a change to <code>myorg/python-lib1</code> and <code>myorg/python-lib2</code> happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.</p>
  <p>I'm looking to restructure the way we build images in <code>myorg/python-app1</code> (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.</p>
  <p>I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.</p>
  
Tags: <docker><continuous-integration><containers><python><testing>
Title: Testing docker image builds when low level libraries change
UserDisplayName: flakes
ThreadText: # Testing docker image builds when low level libraries change
  
  (Asked by: username_0 on 2021-04-10)
  
  **Background**
  
  In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:
  
  * `myorg/python-lib1`: produces package `myorg.lib1`
  * `myorg/python-lib2`: produces package `myorg.lib2`
  * `myorg/python-app`: produces a package `myorg.app`, depends on `myorg.lib1` and `myorg.lib2`
  
  In our CI for `myorg/python-lib1` and `myorg/python-lib1` we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.
  
  In our CI for `myorg/python-app1` we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:
  
  ```
  # Builds image reistry.myorg.com/python-app
  FROM reistry.myorg.com/python-base:latest
  
  COPY . /app
  
  # Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
  RUN pip install /app
  
  ```
  
  ***Note**: We also have some projects using build-packs to achieve a similar goal.*
  
  ---
  
  **Issue**
  
  Because we only run the integration tests for `myorg/python-app1`, we can't get as much confidence when a change to `myorg/python-lib1` and `myorg/python-lib2` happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.
  
  I'm looking to restructure the way we build images in `myorg/python-app1` (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.
  
  I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.
  
  **Tags:** docker, continuous-integration, containers, python, testing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-04-14. Score: 1 | Accepted answer
  
  So I believe I came up with a somewhat re-usable solution for this problem using [devpi](https://pypi.org/project/devpi/). It allows you to use the same Dockerfile for the testing and production image.
  
  Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
  
  Steps to make this work:
  
  **1. Base image from ARG**
  
  I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:
  
  ```
  # Builds image reistry.myorg.com/python-app
  ARG BASE=reistry.myorg.com/python-base:latest
  FROM $BASE
  
  COPY . /app
  
  # Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
  RUN pip install /app
  
  ```
  
  **2. Upload test versions of library**
  
  For each python change:
  
  * Clone the repo
  * `python -m build` the repo
  * `devpi upload` the wheels
  
  **3. Create a layer with devpi pip conf**
  
  Make a pip conf that can access your devpi instance and configure to your index name:
  
  ```
  [global]
  index-url = http://<some ip>:<some port>/root/dev/+simple/
  trusted-host = <some ip>
  disable-pip-version-check = true
  
  [search]
  index = http://<some ip>:<some port>/root/dev/
  
  ```
  
  And add that to your base images pip conf:
  
  ```
  FROM reistry.myorg.com/python-base:latest
  
  COPY pip.conf /etc/pip.conf
  
  ```
  
  **4. Chain the images together**
  
  ```bash
  docker build -t devpi-base -f devpi.Dockerfile .
  docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
  docker tag my-app reistry.myorg.com/my-app:test1
  docker push reistry.myorg.com/my-app:test1
  
  ```
  
  ---
  
  Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!
  
  
  ---
  
  