Answers: <p>You may want to take a look at <a href="https://wiki.jenkins.io/display/JENKINS/ShiningPanda+Plugin" rel="nofollow noreferrer">the ShiningPanda Plugin</a>. </p>
  
  <p>It allows running a build step inside a particular python virtual environment, created automatically if it doesn't exist or reused if it already exists. The virtual environment is located outside the workspace, but it can be named and made to appear as a directory with that name inside the workspace.</p>
  
  <p>To address the dependencies you just have to insert a build step performing the desired <code>pip install -r requirements.txt</code> variant after pulling the workspace (which will probably bring in your <code>requirements.txt</code> file) but before any other build step that would need to use the virtual environment. </p>
  
  <p>The same virtual environment can be used in multiple steps of the same build (by name) and a build can have multiple virtual environments if needed. But AFAIK you can't reuse the same virtual environment in different builds, each build will have its own copy.</p>
  
  <p>The Plugin has a configuration to clear/wipe the environment at any build step using it, if/when needed. Obviously you wouldn't want this always enabled as it would void reusability.</p>
  
  <p>IMHO it works best with <code>requirements.txt</code> files with pinned package versions (like those produced by <a href="https://pypi.org/project/pip-tools/" rel="nofollow noreferrer">pip-tools</a>).</p>
  
  <p><strong>Update</strong>: I recently ran into an issue with the plugin on Windows: I had several jobs in different Jenkins folders but with the same <strong>job names</strong> and same plugin configs (notably the non-empty <strong><code>Name</code></strong> in the <code>Advanced</code> section). Turns out all jobs were actually sharing a single environment (and stepping over each-other), I had to switch to unique values for <code>Name</code> across the jobs. Apparently the 2 names are used to generate the 2 hashes from the virtual environment path, so same names -> same path:</p>
  
  <p><code>\Jenkins\shiningpanda\jobs\&lt;job_name_hash&gt;\virtualenvs\&lt;venv_name_path_hash&gt;\</code></p>
  
  <p>So, if you have the same job names, you actually <strong>can</strong> share the same environment if you really want to :)</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-12-22T22:33:54.497Z
Answers: 2020-01-30T19:30:33.287Z
Answers: 2020-01-30T19:30:33.287Z
Answers: Dan Cornilescu
Answers: <p>I'm going to add more story to the answer @esoterydactyl gave, in case it goes over too many people's heads.</p>
  
  <p>Instead of a monolithic build process, maybe you want to build intermediate packages that can be versioned and deployed to your workspace without all the intermediate build steps. Docker is one packaging solution which is super popular in this case. You could also use .deb or .rpm or non-Docker tarballs, or make CF "StemCells" or even zip files. Also consider Python has prebuilt "wheel" packages now.</p>
  
  <h2>Problems:</h2>
  
  <ul>
  <li>all of your developers must use the same build-deps package for their sandbox or hair-pulling fun will emerge in the differences between dev sandbox and CI workspace. Thus the package must support dev laptop sandboxes and Jenkins workspace installation.</li>
  <li>the build-deps package must have its own CI build/test/packaging/release pipeline. It, however, can have reduced cycle frequency while nobody is committing any changes to it (or any of its components).</li>
  <li>This is an example of bundling. If you have, for example, a CVE to patch, everything that uses the bundle must be patched. This must be no more difficult to deploy than anything else, so don't stop work when your CI builds are fast. CI needs CD, or you could be delivering production undeployable software.</li>
  <li>If you're making your own prebuilt artifacts, you need infrastructure to host the artifact repository, and a versioning scheme.</li>
  </ul>
  
  <p>The incremental solution would be to solve the minimum number of those problems which yields a significant boost in build cycle speed. This is why it makes sense to maintain a local pypi repository, and add Jenkins jobs to build and push wheel packages to it. I know that sounds like a pain, but imagine if you had to manage internal software development for all of the stuff you can just install via pip?</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-10-31T21:49:21.440Z
Answers: 2017-10-31T21:49:21.440Z
Answers: Jeremy
Answers: <p>It's generally a best practice to make your builds idempotent.  Leaving artifacts behind only opens up opportunities for dependency management issues- and resolving those in a resilient way is exactly why you're using a build server.  I suggest you take a look at WHY your Python builds are running so slow- if you discover this is because of an overabundance of packages in each project, this might be optimizable.  If you discover this is because of server performance issues, those are easily fixable with more resources.  If you find you are blocked by network throughput, consider using an artifact repository to cache packages locally.  But building persistent virtualenvs seems like a recipe for problems that could be unpredictable when moving to another server, and extremely frustrating to troubleshoot.  </p>
  
Answers: CC BY-SA 3.0
Answers: 2017-10-28T02:21:22.880Z
Answers: 2017-10-28T02:21:22.880Z
Answers: esoterydactyl
Body: <p>Currently, every time we run a build through Jenkins+Ansible, we are re-creating a virtual environment and re-installing all the dependencies  listed inside the <code>requirements.txt</code> file.</p>
  
  <p>This is very slow and does not scale well. How can we improve and speed up the process? Can we re-use virtual environments?</p>
  
  <hr>
  
  <p>Our latest idea which we have not yet implemented was to build virtual environments outside of a Jenkins workspace, name virtual environments based on a project+branch and keep an MD5 sum of the requirements for every virtual environment. Then, before building an environment the next time, we calculate the MD5 sum of the current requirements in this branch and look up existing virtual environments by this MD5 sum. If an existing environment with this sum was found, just re-use it. We are not sure if this is the best way to solve the problem.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-09-12T23:59:03.733Z
Comments: It's not a virtualenv-specific solution, but you might consider running builds in a docker or vagrant image that has the virtualenv pre-installed and pre-configured.
Comments: jayhendren
Tags: <jenkins><ansible><python><virtualenv>
Title: Re-using Python virtual environments on a build server
UserDisplayName: alecxe
ThreadText: # Re-using Python virtual environments on a build server
  
  (Asked by: username_0 on 2017-09-12)
  
  Currently, every time we run a build through Jenkins+Ansible, we are re-creating a virtual environment and re-installing all the dependencies listed inside the `requirements.txt` file.
  
  This is very slow and does not scale well. How can we improve and speed up the process? Can we re-use virtual environments?
  
  ---
  
  Our latest idea which we have not yet implemented was to build virtual environments outside of a Jenkins workspace, name virtual environments based on a project+branch and keep an MD5 sum of the requirements for every virtual environment. Then, before building an environment the next time, we calculate the MD5 sum of the current requirements in this branch and look up existing virtual environments by this MD5 sum. If an existing environment with this sum was found, just re-use it. We are not sure if this is the best way to solve the problem.
  
  **Tags:** jenkins, ansible, python, virtualenv
  
  > **Comments:**
  > **username_1**: It's not a virtualenv-specific solution, but you might consider running builds in a docker or vagrant image that has the virtualenv pre-installed and pre-configured.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2017-10-28. Score: 5 | Accepted answer
  
  It's generally a best practice to make your builds idempotent. Leaving artifacts behind only opens up opportunities for dependency management issues- and resolving those in a resilient way is exactly why you're using a build server. I suggest you take a look at WHY your Python builds are running so slow- if you discover this is because of an overabundance of packages in each project, this might be optimizable. If you discover this is because of server performance issues, those are easily fixable with more resources. If you find you are blocked by network throughput, consider using an artifact repository to cache packages locally. But building persistent virtualenvs seems like a recipe for problems that could be unpredictable when moving to another server, and extremely frustrating to troubleshoot.
  
  
  ---
  
  ### Answer by username_2 on 2019-12-22. Score: 3
  
  You may want to take a look at [the ShiningPanda Plugin](https://wiki.jenkins.io/display/JENKINS/ShiningPanda+Plugin).
  
  It allows running a build step inside a particular python virtual environment, created automatically if it doesn't exist or reused if it already exists. The virtual environment is located outside the workspace, but it can be named and made to appear as a directory with that name inside the workspace.
  
  To address the dependencies you just have to insert a build step performing the desired `pip install -r requirements.txt` variant after pulling the workspace (which will probably bring in your `requirements.txt` file) but before any other build step that would need to use the virtual environment.
  
  The same virtual environment can be used in multiple steps of the same build (by name) and a build can have multiple virtual environments if needed. But AFAIK you can't reuse the same virtual environment in different builds, each build will have its own copy.
  
  The Plugin has a configuration to clear/wipe the environment at any build step using it, if/when needed. Obviously you wouldn't want this always enabled as it would void reusability.
  
  IMHO it works best with `requirements.txt` files with pinned package versions (like those produced by [pip-tools](https://pypi.org/project/pip-tools/)).
  
  **Update**: I recently ran into an issue with the plugin on Windows: I had several jobs in different Jenkins folders but with the same **job names** and same plugin configs (notably the non-empty **`Name`** in the `Advanced` section). Turns out all jobs were actually sharing a single environment (and stepping over each-other), I had to switch to unique values for `Name` across the jobs. Apparently the 2 names are used to generate the 2 hashes from the virtual environment path, so same names -> same path:
  
  `\Jenkins\shiningpanda\jobs\<job_name_hash>\virtualenvs\<venv_name_path_hash>\`
  
  So, if you have the same job names, you actually **can** share the same environment if you really want to :)
  
  
  ---
  
  ### Answer by username_3 on 2017-10-31. Score: 3
  
  I'm going to add more story to the answer @username_4 gave, in case it goes over too many people's heads.
  
  Instead of a monolithic build process, maybe you want to build intermediate packages that can be versioned and deployed to your workspace without all the intermediate build steps. Docker is one packaging solution which is super popular in this case. You could also use .deb or .rpm or non-Docker tarballs, or make CF "StemCells" or even zip files. Also consider Python has prebuilt "wheel" packages now.
  
  ## Problems:
  
  * all of your developers must use the same build-deps package for their sandbox or hair-pulling fun will emerge in the differences between dev sandbox and CI workspace. Thus the package must support dev laptop sandboxes and Jenkins workspace installation.
  * the build-deps package must have its own CI build/test/packaging/release pipeline. It, however, can have reduced cycle frequency while nobody is committing any changes to it (or any of its components).
  * This is an example of bundling. If you have, for example, a CVE to patch, everything that uses the bundle must be patched. This must be no more difficult to deploy than anything else, so don't stop work when your CI builds are fast. CI needs CD, or you could be delivering production undeployable software.
  * If you're making your own prebuilt artifacts, you need infrastructure to host the artifact repository, and a versioning scheme.
  
  The incremental solution would be to solve the minimum number of those problems which yields a significant boost in build cycle speed. This is why it makes sense to maintain a local pypi repository, and add Jenkins jobs to build and push wheel packages to it. I know that sounds like a pain, but imagine if you had to manage internal software development for all of the stuff you can just install via pip?
  
  
  ---
  
  