Answers: <p>To me, but correct me if I am wrong, the Jenkins servers should not be rebooted, but they should create VMs and reboot them. When I read the question I get the impression the slaves are rebooted, i.e. <code>I can't just use the executor on the slave node because we reboot the server and that would kill the job.</code> To me it seems that the jenkins slave should create a VM, run the tests, reboot the system, run some additional tests and finally remove the VMs.</p>
  
  <p>From a production perspective, but again correct me if I am wrong the team is not using Jenkins to provision the systems and subsequently reboot the nodes? It could be possible that they use Jenkins to start the provision step, but they will not reboot the Jenkins systems right?</p>
  
  <p>I would recommend to start with the Production perspective, e.g. How does the team provision the systems at the moment? E.g. Chef? If that is the case, how to they run that? If they would run chef apply then this could be run by a Jenkins slave, but I would not reboot the jenkins slave but automate the process of creating a VM, provision it, reboot it and kill it. The latter is only applicable for testing purposes.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-12-04T02:00:07.707Z
Answers: "automate the process of creating a VM, provision it, reboot it and kill it" -- I absolutely agree, but unfortunately, some of our infrastructure providers do not give us this ability and testing for the pets in their infrastructure is the issue.
Answers: mghicks
Answers: CC BY-SA 3.0
Answers: 2017-12-03T13:55:04.743Z
Answers: 2017-12-03T13:55:04.743Z
Answers: 030
Answers: <p>I found the missing piece in the <a href="https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin" rel="nofollow noreferrer">Lockable resource plugin</a>.</p>
  
  <pre><code>lock(label: 'server-pool', quantity: 1) {
     ...
  }
  </code></pre>
  
  <p>Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-08T20:43:51.287Z
Answers: 2018-02-08T20:43:51.287Z
Answers: mghicks
Body: <p>I have a pool of servers to test a provisioning process my team owns.  One server is needed per branch or PR.  The process requires reboots.  Once the process is run, we run some tests.  A Jenkins pipeline automates the provisioning process and test.</p>
  
  <p>So my problem is how to manage the pool of servers so that </p>
  
  <ol>
  <li>Any available pool server can be selected for a build</li>
  <li>A server claimed for a build is unavailable for other builds</li>
  <li>Upon successful build, the server is made available again</li>
  <li>Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)</li>
  </ol>
  
  <p>I can't just use the executor on the slave node because we reboot the server and that would kill the job.</p>
  
  <p>My current solution is:</p>
  
  <ol>
  <li>Pool servers are setup as Jenkins slaves with one executor each and a common label</li>
  <li>A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves <code>slave.jar</code> to protect against Jenkins restarting the assigned slave (which it annoyingly will do)</li>
  <li>The next pipeline stage uses <code>node.getComputer().disconnect()</code> to disable the slave--it seems this must run on master so must be in a different stage</li>
  <li>A <code>post { success { ... } }</code> uses ssh to move <code>slave.jar</code> back into place on the server and <code>node.getComputer().connect(true)</code> to allow it to be picked again</li>
  </ol>
  
  <p>This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 &amp; 3.  When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave.  When this happens, things break badly.</p>
  
  <p>Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?</p>
  
  <p>Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run?  Plugins I've missed, etc.</p>
  
  <p>I've examined the few results google returns for the slave disconnect problem and the most promising one was <a href="https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw" rel="nofollow noreferrer">https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw</a> -- the suggestion there failed for me because <code>OfflineCause.UserCause</code> class could not be resolved in my pipeline.  I only know enough Java and Groovy to be dangerous!</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-11-23T04:01:59.363Z
Comments: Seems like you're using Jenkins like a psuedo-hypervisor.  Are you using public cloud, private cloud, or bare metal? You may have better luck managing a pool of systems using the API for your cloud provider.
Comments: jayhendren
Comments: CC BY-SA 3.0
Comments: 2017-11-23T13:22:32.727Z
Comments: @jayhendren we do call into API's for our IaaS providers as those are part of what we have to test--but, we don't have the ability to spin up servers on demand, sadly.
Comments: mghicks
Comments: CC BY-SA 3.0
Comments: 2017-12-03T13:43:40.633Z
Comments: `provisioning process` Should I think of a CMS, e.g. puppet, ansible?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-12-04T02:01:50.143Z
Comments: @030 yes, ansible specifically.
Comments: mghicks
Tags: <jenkins><jenkins-pipeline>
Title: Jenkins managing a pool of servers across restarts
UserDisplayName: mghicks
ThreadText: # Jenkins managing a pool of servers across restarts
  
  (Asked by: username_0 on 2017-11-22)
  
  I have a pool of servers to test a provisioning process my team owns. One server is needed per branch or PR. The process requires reboots. Once the process is run, we run some tests. A Jenkins pipeline automates the provisioning process and test.
  
  So my problem is how to manage the pool of servers so that
  
  1. Any available pool server can be selected for a build
  2. A server claimed for a build is unavailable for other builds
  3. Upon successful build, the server is made available again
  4. Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)
  
  I can't just use the executor on the slave node because we reboot the server and that would kill the job.
  
  My current solution is:
  
  1. Pool servers are setup as Jenkins slaves with one executor each and a common label
  2. A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves `slave.jar` to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
  3. The next pipeline stage uses `node.getComputer().disconnect()` to disable the slave--it seems this must run on master so must be in a different stage
  4. A `post { success { ... } }` uses ssh to move `slave.jar` back into place on the server and `node.getComputer().connect(true)` to allow it to be picked again
  
  This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 & 3. When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave. When this happens, things break badly.
  
  Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?
  
  Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run? Plugins I've missed, etc.
  
  I've examined the few results google returns for the slave disconnect problem and the most promising one was <https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw> -- the suggestion there failed for me because `OfflineCause.UserCause` class could not be resolved in my pipeline. I only know enough Java and Groovy to be dangerous!
  
  **Tags:** jenkins, jenkins-pipeline
  
  > **Comments:**
  > **username_1**: Seems like you're using Jenkins like a psuedo-hypervisor.  Are you using public cloud, private cloud, or bare metal? You may have better luck managing a pool of systems using the API for your cloud provider.  
  > **username_0**: @username_1 we do call into API's for our IaaS providers as those are part of what we have to test--but, we don't have the ability to spin up servers on demand, sadly.  
  > **username_2**: `provisioning process` Should I think of a CMS, e.g. puppet, ansible?  
  > **username_0**: @username_2 yes, ansible specifically.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-02-08. Score: 4 | Accepted answer
  
  I found the missing piece in the [Lockable resource plugin](https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin).
  
  ```
  lock(label: 'server-pool', quantity: 1) {
     ...
  }
  
  ```
  
  Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.
  
  
  ---
  
  ### Answer by username_2 on 2017-12-03. Score: 3
  
  To me, but correct me if I am wrong, the Jenkins servers should not be rebooted, but they should create VMs and reboot them. When I read the question I get the impression the slaves are rebooted, i.e. `I can't just use the executor on the slave node because we reboot the server and that would kill the job.` To me it seems that the jenkins slave should create a VM, run the tests, reboot the system, run some additional tests and finally remove the VMs.
  
  From a production perspective, but again correct me if I am wrong the team is not using Jenkins to provision the systems and subsequently reboot the nodes? It could be possible that they use Jenkins to start the provision step, but they will not reboot the Jenkins systems right?
  
  I would recommend to start with the Production perspective, e.g. How does the team provision the systems at the moment? E.g. Chef? If that is the case, how to they run that? If they would run chef apply then this could be run by a Jenkins slave, but I would not reboot the jenkins slave but automate the process of creating a VM, provision it, reboot it and kill it. The latter is only applicable for testing purposes.
  
  > **Comments:**
  > **username_0**: "automate the process of creating a VM, provision it, reboot it and kill it" -- I absolutely agree, but unfortunately, some of our infrastructure providers do not give us this ability and testing for the pets in their infrastructure is the issue.  
  
  ---
  
  