Answers: <p>I use <a href="https://www.theforeman.org/" rel="nofollow noreferrer">The Foreman</a> for this. It leverages the <a href="https://github.com/fog/fog-vsphere" rel="nofollow noreferrer">fog vsphere</a> module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.</p>
  
  <p>This software stack is backed by Red Hat and re-branded as <a href="https://access.redhat.com/products/red-hat-satellite" rel="nofollow noreferrer">Red Hat Satellite 6</a> if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.</p>
  
  <p>Upon deployment of your VM, you have the following options for configuring your newly deployed VM:</p>
  
  <ul>
  <li>Kickstart script</li>
  <li>Puppet</li>
  <li>Ansible</li>
  <li>Chef</li>
  <li>SaltStack</li>
  </ul>
  
  <p>As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.</p>
  
  <p>This software comes with a <a href="https://theforeman.org/api/1.17/" rel="nofollow noreferrer">REST API</a> that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-03-22T15:44:16.940Z
Answers: 2018-03-22T15:44:16.940Z
Answers: James Shewey
Answers: <p>Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.</p>
  
  <p>I would further split your two Tasks a bit more differently, such that Step 1 would be to just "<em>Set up the VM</em>", and Step 2 would be to "<em>Provision the VM with required programs and configuration</em>".</p>
  
  <p>My 2-cents would be to use tools such as <a href="https://www.packer.io/" rel="nofollow noreferrer">Packer</a> and <a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a>.</p>
  
  <p>You would use (for example), the <a href="https://www.packer.io/docs/builders/virtualbox-ovf.html" rel="nofollow noreferrer">VirtualBox Packer builder</a>, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the <a href="https://www.packer.io/docs/provisioners/ansible-local.html" rel="nofollow noreferrer">Ansible-local</a> provisioner, or something to this effect, to run a <a href="https://docs.ansible.com/ansible/latest/playbooks.html" rel="nofollow noreferrer">playbook</a>, which would handle the rest of Step 1, and all of Step 2.</p>
  
  <hr>
  
  <p>The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.</p>
  
  <p>There are of course, other tools such as <a href="https://www.vagrantup.com/" rel="nofollow noreferrer">Vagrant</a>, <a href="https://puppet.com/" rel="nofollow noreferrer">Puppet</a>, and <a href="https://www.chef.io/chef/" rel="nofollow noreferrer">Chef</a>, which can handle what you'd like to achieve, so it's up to you find your preferred tool.</p>
  
  <hr>
  
  <p>Mini breakdown of Ansible modules you can use:</p>
  
  <ul>
  <li>Open up port: <a href="https://docs.ansible.com/ansible/latest/win_firewall_rule_module.html" rel="nofollow noreferrer">win_firewal_rule_module</a></li>
  <li>Installing bitvise and python: <a href="https://docs.ansible.com/ansible/latest/win_chocolatey_module.html" rel="nofollow noreferrer">win_chocolatey</a></li>
  <li>Setting up pip and virtual envs: <a href="https://docs.ansible.com/ansible/latest/pip_module.html" rel="nofollow noreferrer">pip</a></li>
  <li>Transferring public keys / requirements.txt etc: <a href="https://docs.ansible.com/ansible/latest/win_copy_module.html" rel="nofollow noreferrer">win_copy</a></li>
  </ul>
  
  <p>Etc. The full list of Ansible Windows Modules can be found <a href="https://docs.ansible.com/ansible/latest/list_of_windows_modules.html" rel="nofollow noreferrer">here</a>.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-03-20T10:50:02.460Z
Answers: 2018-03-20T10:50:02.460Z
Answers: Rekovni
Body: <h2>Automating creating of new VM and pushing code to server</h2>
  
  <p>Hi all,</p>
  
  <p>I am writing a integration test for a client-server application. To simulate
  the production server setup, and to update the code, I set up VMs on my
  computer. The application runs on a Windows server, so I have to use systems
  compatible with Windows. </p>
  
  <h2>STEP 1: Set up VM properly:</h2>
  
  <ol>
  <li>Download the OVA file from MS</li>
  <li>create a new VM from the OVA file</li>
  <li>reset the mac id</li>
  <li>set a static dns ip address and a VM ip address</li>
  <li>open up the port 22</li>
  <li>install bitvise ssh server</li>
  <li>transfer public keys to the VM</li>
  <li>install python 3.5.2</li>
  <li>update <code>pip</code></li>
  <li>install <code>virtualenv</code> systemwide</li>
  </ol>
  
  <h3>STEP 2: Deploy code to VM:</h3>
  
  <ol>
  <li>create a requirements file using pip using: <code>pip freeze -r requirements</code></li>
  <li>zip the entire repo using powershell using: 
  <code>powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"</code></li>
  <li>using <code>pscp</code> I transfer the files to the VM using:
  <code>pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop</code> </li>
  <li>unzip the files again using powershell using:
  <code>powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"</code></li>
  <li>delete the <code>virtualenv</code> folder called <code>env</code> in the folder using <code>plink</code></li>
  <li>create a fresh <code>env</code> folder using using <code>virtualenv</code>: <code>virtualenv env</code></li>
  <li>using the <code>requirements.txt</code> document, I install the requirements again by
  direcatly using the <code>pip</code> installed in the virtualenv above using:
  <code>/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt</code></li>
  <li>run the file with the <code>python</code> distribution installed in the virtualenv
  using:
  <code>/path/to/python file_name.py</code></li>
  </ol>
  
  <p>Because I was trying to execute relatively complex commands, which in turn had
  multiple quotes (double and single), spaces etc, I had to spend quite a bit of
  time just debbuging how to run these commands over the network.  I have
  automated most of the deployment part(<em>STEP 2</em>) using a combination of
  <code>subprocess.call</code>, <code>plink</code>, and <code>pscp</code>. However, I haven't yet automated
  setting up the VM (<em>STEP 1</em>).</p>
  
  <p>I am guessing, the above set of actions are not just something I am doing, and
  probably others have done this as well. </p>
  
  <p>Currently, I am using low level tools to acheive this. While this works for
  setting up Windows VM, it wont probably work for Linux based VMs or if I change
  python versions use a different language etc etc.</p>
  
  <p>I have also searched a bit and came across this book:
  <a href="https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html" rel="nofollow noreferrer">https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html</a></p>
  
  <p>where the author goes into how to deploy to a linux server. However, it seems
  that he is essentially doing what I am doing by directly running commands on
  the linux server, just through the <code>fabric</code> wrapper. Because <code>ssh</code> doesn't work
  as well on Windows, I am guessing, I will have to get back to the nitty-grittys
  of deployment with <code>fabric</code> as well.</p>
  
  <h3>Question:</h3>
  
  <ol>
  <li><p>Is there a better way to setup the VM, and deploy code to the VMs, while
  abstracting away the difference between different operating systems, python
  versions, tools etc?</p></li>
  <li><p>I have also heard of a <code>git</code> based workflow. Is that better than directly pushing the code to the server? Why so?</p></li>
  </ol>
  
Comments: CC BY-SA 3.0
Comments: 2018-03-20T09:28:44.983Z
Comments: When looking for this kind of things I've choosen Chef because the client has native support for windows functionality and there's a cookbook for python supporting windows made by coderanger which is someone you can be confident in the quality of work: https://github.com/poise/poise-python#windows-support
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2018-03-20T09:30:46.797Z
Comments: BTW I'm afraid this is going to be mainly opinion based, everyone will  have his/her own method working great for him/her. You mainly need a mix of Infrastructure as code and configuration management. I think vagrant+chef could be a good setup for your case, but there's probably other.
Comments: Tensibai
Tags: <deployment><git><python><virtualenv>
Title: Automating creation of new VM and pushing code to server
UserDisplayName: alpha_989
ThreadText: # Automating creation of new VM and pushing code to server
  
  (Asked by: username_0 on 2018-03-19)
  
  ## Automating creating of new VM and pushing code to server
  
  Hi all,
  
  I am writing a integration test for a client-server application. To simulate
  the production server setup, and to update the code, I set up VMs on my
  computer. The application runs on a Windows server, so I have to use systems
  compatible with Windows.
  
  ## STEP 1: Set up VM properly:
  
  1. Download the OVA file from MS
  2. create a new VM from the OVA file
  3. reset the mac id
  4. set a static dns ip address and a VM ip address
  5. open up the port 22
  6. install bitvise ssh server
  7. transfer public keys to the VM
  8. install python 3.5.2
  9. update `pip`
  10. install `virtualenv` systemwide
  
  ### STEP 2: Deploy code to VM:
  
  1. create a requirements file using pip using: `pip freeze -r requirements`
  2. zip the entire repo using powershell using:
     `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
  3. using `pscp` I transfer the files to the VM using:
     `pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop`
  4. unzip the files again using powershell using:
     `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
  5. delete the `virtualenv` folder called `env` in the folder using `plink`
  6. create a fresh `env` folder using using `virtualenv`: `virtualenv env`
  7. using the `requirements.txt` document, I install the requirements again by
     direcatly using the `pip` installed in the virtualenv above using:
     `/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt`
  8. run the file with the `python` distribution installed in the virtualenv
     using:
     `/path/to/python file_name.py`
  
  Because I was trying to execute relatively complex commands, which in turn had
  multiple quotes (double and single), spaces etc, I had to spend quite a bit of
  time just debbuging how to run these commands over the network. I have
  automated most of the deployment part(*STEP 2*) using a combination of
  `subprocess.call`, `plink`, and `pscp`. However, I haven't yet automated
  setting up the VM (*STEP 1*).
  
  I am guessing, the above set of actions are not just something I am doing, and
  probably others have done this as well.
  
  Currently, I am using low level tools to acheive this. While this works for
  setting up Windows VM, it wont probably work for Linux based VMs or if I change
  python versions use a different language etc etc.
  
  I have also searched a bit and came across this book:
  <https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html>
  
  where the author goes into how to deploy to a linux server. However, it seems
  that he is essentially doing what I am doing by directly running commands on
  the linux server, just through the `fabric` wrapper. Because `ssh` doesn't work
  as well on Windows, I am guessing, I will have to get back to the nitty-grittys
  of deployment with `fabric` as well.
  
  ### Question:
  
  1. Is there a better way to setup the VM, and deploy code to the VMs, while
     abstracting away the difference between different operating systems, python
     versions, tools etc?
  2. I have also heard of a `git` based workflow. Is that better than directly pushing the code to the server? Why so?
  
  **Tags:** deployment, git, python, virtualenv
  
  > **Comments:**
  > **username_1**: When looking for this kind of things I've choosen Chef because the client has native support for windows functionality and there's a cookbook for python supporting windows made by coderanger which is someone you can be confident in the quality of work: https://github.com/poise/poise-python#windows-support  
  > **username_1**: BTW I'm afraid this is going to be mainly opinion based, everyone will  have his/her own method working great for him/her. You mainly need a mix of Infrastructure as code and configuration management. I think vagrant+chef could be a good setup for your case, but there's probably other.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-03-22. Score: 2
  
  I use [The Foreman](https://www.theforeman.org/) for this. It leverages the [fog vsphere](https://github.com/fog/fog-vsphere) module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.
  
  This software stack is backed by Red Hat and re-branded as [Red Hat Satellite 6](https://access.redhat.com/products/red-hat-satellite) if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.
  
  Upon deployment of your VM, you have the following options for configuring your newly deployed VM:
  
  * Kickstart script
  * Puppet
  * Ansible
  * Chef
  * SaltStack
  
  As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.
  
  This software comes with a [REST API](https://theforeman.org/api/1.17/) that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).
  
  
  ---
  
  ### Answer by username_3 on 2018-03-20. Score: 2
  
  Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what username_1 has suggested in his comments.
  
  I would further split your two Tasks a bit more differently, such that Step 1 would be to just "*Set up the VM*", and Step 2 would be to "*Provision the VM with required programs and configuration*".
  
  My 2-cents would be to use tools such as [Packer](https://www.packer.io/) and [Ansible](https://www.ansible.com/).
  
  You would use (for example), the [VirtualBox Packer builder](https://www.packer.io/docs/builders/virtualbox-ovf.html), to stand up your VM, which would handle points 1-4 of your Step 1, and then use the [Ansible-local](https://www.packer.io/docs/provisioners/ansible-local.html) provisioner, or something to this effect, to run a [playbook](https://docs.ansible.com/ansible/latest/playbooks.html), which would handle the rest of Step 1, and all of Step 2.
  
  ---
  
  The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.
  
  There are of course, other tools such as [Vagrant](https://www.vagrantup.com/), [Puppet](https://puppet.com/), and [Chef](https://www.chef.io/chef/), which can handle what you'd like to achieve, so it's up to you find your preferred tool.
  
  ---
  
  Mini breakdown of Ansible modules you can use:
  
  * Open up port: [win\_firewal\_rule\_module](https://docs.ansible.com/ansible/latest/win_firewall_rule_module.html)
  * Installing bitvise and python: [win\_chocolatey](https://docs.ansible.com/ansible/latest/win_chocolatey_module.html)
  * Setting up pip and virtual envs: [pip](https://docs.ansible.com/ansible/latest/pip_module.html)
  * Transferring public keys / requirements.txt etc: [win\_copy](https://docs.ansible.com/ansible/latest/win_copy_module.html)
  
  Etc. The full list of Ansible Windows Modules can be found [here](https://docs.ansible.com/ansible/latest/list_of_windows_modules.html).
  
  
  ---
  
  