Answers: <blockquote>
    <p>What are the general ways to tackle this problem?</p>
  </blockquote>
  
  <p>The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.</p>
  
  <p>Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:</p>
  
  <pre><code>python3.6 -m venv --copies venv
  sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&amp;\&amp; pwd)"/' venv/bin/activate
  sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*
  </code></pre>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-17T08:38:56.583Z
Answers: 2018-01-22T05:33:34.667Z
Answers: 2018-01-22T05:33:34.667Z
Answers: user2640621
Body: <p>Currently, we build our application on a build server where we create a virtual environment using <a href="https://virtualenv.pypa.io/en/stable/" rel="nofollow noreferrer"><code>virtualenv</code> command</a>, install all the Python dependencies into it, then "patch" it using the following bash commands to make it <em>relocatable</em>:</p>
  
  <pre><code># Adopt virtualenv to the target system
  echo "* Patching virtual envirinment for target system"
  for f in "${VENV_PATH}/bin"/* ; do
      if [ -n "$(file "${f}" | grep "text")" ] ; then
      echo "  + ${f}"
          sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
      fi
  done
  rm -rf "${VENV_PATH}/lib64"
  ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"
  </code></pre>
  
  <p>Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via <code>yum</code>.</p>
  
  <p>The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to <em>avoid installing the same dependencies on all the target servers</em>.</p>
  
  <p>The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).</p>
  
  <p>Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate <code>pip</code> installation commands with, for example, <a href="https://github.com/paylogic/pip-accel" rel="nofollow noreferrer"><code>pip-accel</code></a> or caching) What are the general ways to tackle this problem?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-09-10T00:21:52.097Z
Comments: unfortunately the `--relocatable` flag for virtualenv never did work well, as documented here
  
  https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable
  
  If the project found it fragile I bet it will be for you too.
  
  deb's have `dh-virtualenv` but I don't know of one for rpm, but how about pip freeze and wheel?
  
  `pip wheel --wheel-dir=./artifacts -r requirements.txt`
  
  Package ./artifacts into your RPM, create and activate the venv then run `pip install ./artifacts/`
  
  We shove our wheels into a gcp bucket and just add the urls for builds via jenkins.
Comments: gdahlm
Tags: <deployment><python><performance><virtualenv>
Title: Re-using Python virtual environment vs Re-building it
UserDisplayName: alecxe
ThreadText: # Re-using Python virtual environment vs Re-building it
  
  (Asked by: username_0 on 2017-09-07)
  
  Currently, we build our application on a build server where we create a virtual environment using [`virtualenv` command](https://virtualenv.pypa.io/en/stable/), install all the Python dependencies into it, then "patch" it using the following bash commands to make it *relocatable*:
  
  ```
  # Adopt virtualenv to the target system
  echo "* Patching virtual envirinment for target system"
  for f in "${VENV_PATH}/bin"/* ; do
      if [ -n "$(file "${f}" | grep "text")" ] ; then
      echo "  + ${f}"
          sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
      fi
  done
  rm -rf "${VENV_PATH}/lib64"
  ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"
  
  ```
  
  Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via `yum`.
  
  The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to *avoid installing the same dependencies on all the target servers*.
  
  The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).
  
  Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate `pip` installation commands with, for example, [`pip-accel`](https://github.com/paylogic/pip-accel) or caching) What are the general ways to tackle this problem?
  
  **Tags:** deployment, python, performance, virtualenv
  
  > **Comments:**
  > **username_1**: unfortunately the `--relocatable` flag for virtualenv never did work well, as documented here  https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable  If the project found it fragile I bet it will be for you too.  deb's have `dh-virtualenv` but I don't know of one for rpm, but how about pip freeze and wheel?  `pip wheel --wheel-dir=./artifacts -r requirements.txt`  Package ./artifacts into your RPM, create and activate the venv then run `pip install ./artifacts/`  We shove our wheels into a gcp bucket and just add the urls for builds via jenkins.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-09-17. Score: 4 | Accepted answer
  
  > What are the general ways to tackle this problem?
  
  The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.
  
  Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:
  
  ```
  python3.6 -m venv --copies venv
  sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
  sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*
  
  ```
  
  
  ---
  
  