Answers: <p>Quick answer:</p>
  <p>Changing <code>sudo reboot</code>  to <code>sudo shutdown -r +m5</code> or what ever time delay needed, would prevent the pipeline from failing.  It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.</p>
  <p>Longer (preventing potential conflicts answer):</p>
  <p>Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.</p>
  <p>If the agents are configured to launch via ssh,</p>
  <p><a href="https://i.sstatic.net/CCTQa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CCTQa.png" alt="launch agents via ssh" /></a></p>
  <p>and configured to be brought offline if not in use,</p>
  <p><a href="https://i.sstatic.net/fiCz7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fiCz7.png" alt="take offline when idle" /></a></p>
  <p>Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.</p>
  <p>From the man page for shutdown (on one of my servers)</p>
  <pre><code>If the time argument is used, 5 minutes before the system goes down the
  /run/nologin file is created to ensure that further logins shall not be
  allowed.
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2022-07-07T03:26:30.157Z
Answers: *Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.* Wow, I had no idea `shutdown` could prevent logins. Manual testing indicates this might work. I'll test this out in CI to confirm.
Answers: Sagar
Answers: CC BY-SA 4.0
Answers: 2022-07-07T21:59:21.500Z
Answers: unfortunately the agent remains logged in if there is another pipeline already queued. So, the next pipeline starts, and is abruptly killed 5 minutes into the job. Is there a way to force each pipeline to use a new agent?
Answers: Sagar
Answers: CC BY-SA 4.0
Answers: 2022-07-06T14:30:42.673Z
Answers: 2022-07-06T14:30:42.673Z
Answers: John
Answers: <p>Some ideas to get you started:</p>
  <ul>
  <li><p>Create a separate job to shut down the node and call that with a <a href="https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/" rel="nofollow noreferrer">build step</a> from your pipeline with &quot;wait: false&quot;, e.g.: <code>build(job: &quot;shutdown_host&quot;, wait: false)</code>.  The &quot;shutdown_host&quot; job would probably need to have a sleep statement or something before shutting down the node to avoid killing the node before the parent job has a chance to exit.</p>
  </li>
  <li><p>Execute a short shell script that forks &amp; detaches a new process to shut down the host.  Again you would probably need a sleep statement in the forked process to avoid shutting down the node before the Jenkins job exits.  Also, in this case, you would need to disable the <a href="https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller" rel="nofollow noreferrer">Process Tree Killer</a> to avoid the forked process being killed.</p>
  </li>
  <li><p>Execute a step on the Jenkins master to ssh into your node to shut it down.  But this would depend on the Jenkins master having ssh access to your node and administrative privileges to shut it down.</p>
  </li>
  </ul>
  <p>I'm sure there are some other clever and creative ideas that I'm missing as well.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-06-09T05:25:34.107Z
Answers: Isn't there a race in all of these solutions? Suppose one job is executing, and a second is queued. If the second job starts while the first is sleeping in the background, then won't the second job be disrupted due to the reboot?
Answers: Sagar
Answers: CC BY-SA 4.0
Answers: 2022-06-09T18:36:25.000Z
Answers: Yes, there will be.  You can check in your pipeline code if there are any jobs running on a node (there should be other Stack Exchange questions that address how to do this) before rebooting, or you can use [resource locks](https://www.jenkins.io/doc/pipeline/steps/lockable-resources/) (which are kind of like semaphores) to prevent race conditions.
Answers: jayhendren
Answers: CC BY-SA 4.0
Answers: 2022-06-03T15:44:24.610Z
Answers: 2022-06-03T15:44:24.610Z
Answers: jayhendren
Body: <p>I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.</p>
  <p>Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).</p>
  <p>To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:</p>
  <pre><code>+ sudo reboot
  Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel &quot;hudson.remoting.Channel@35ca6b21: baremetal&quot;: Remote call on baremetal failed. The channel is closing down or has closed down
  wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
  (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  ERROR: script returned exit code -1
  Setting status of *** to FAILURE with url *** and message: ' '
  Using context: Bare Metal Benchmarks
  Finished: FAILURE
  </code></pre>
  <p>The next instance of this pipeline can then start running on the rebooted bare metal node.</p>
  <p>Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?</p>
  <p>The basic structure of the scripted pipeline is:</p>
  <pre class="lang-java prettyprint-override"><code>node('baremetal') {
      checkout scm
  
      try { 
          stage('Benchmarks') {
              // run benchmarks
          }
  
          stage('Process data') {
              // process data and send results to users
          }
      }
      finally {
          stage('Clean up') {
              sh 'sudo reboot'
          }
      }
  }
  </code></pre>
  
Comments: CC BY-SA 4.0
Comments: 2023-03-21T22:10:49.750Z
Comments: Sagar, what was your solution for this? I have the same use-case.
Comments: Ungeheuer
Comments: CC BY-SA 4.0
Comments: 2023-03-21T23:29:19.330Z
Comments: @Ungeheuer unfortunately, I haven't found a robust solution. Please do post an answer if you find one. Thanks!
Comments: Sagar
Tags: <jenkins><jenkins-pipeline><linux>
Title: Rebooting node at end of Jenkins pipeline
UserDisplayName: Sagar
ThreadText: # Rebooting node at end of Jenkins pipeline
  
  (Asked by: username_0 on 2022-06-02)
  
  I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.
  
  Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).
  
  To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:
  
  ```
  + sudo reboot
  Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@35ca6b21: baremetal": Remote call on baremetal failed. The channel is closing down or has closed down
  wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
  (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  ERROR: script returned exit code -1
  Setting status of *** to FAILURE with url *** and message: ' '
  Using context: Bare Metal Benchmarks
  Finished: FAILURE
  
  ```
  
  The next instance of this pipeline can then start running on the rebooted bare metal node.
  
  Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?
  
  The basic structure of the scripted pipeline is:
  
  ```java
  node('baremetal') {
      checkout scm
  
      try { 
          stage('Benchmarks') {
              // run benchmarks
          }
  
          stage('Process data') {
              // process data and send results to users
          }
      }
      finally {
          stage('Clean up') {
              sh 'sudo reboot'
          }
      }
  }
  
  ```
  
  **Tags:** jenkins, jenkins-pipeline, linux
  
  > **Comments:**
  > **username_1**: username_0, what was your solution for this? I have the same use-case.  
  > **username_0**: @username_1 unfortunately, I haven't found a robust solution. Please do post an answer if you find one. Thanks!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-06. Score: 1
  
  Quick answer:
  
  Changing `sudo reboot` to `sudo shutdown -r +m5` or what ever time delay needed, would prevent the pipeline from failing. It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.
  
  Longer (preventing potential conflicts answer):
  
  Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.
  
  If the agents are configured to launch via ssh,
  
  [![launch agents via ssh](https://i.sstatic.net/CCTQa.png)](https://i.sstatic.net/CCTQa.png)
  
  and configured to be brought offline if not in use,
  
  [![take offline when idle](https://i.sstatic.net/fiCz7.png)](https://i.sstatic.net/fiCz7.png)
  
  Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.
  
  From the man page for shutdown (on one of my servers)
  
  ```
  If the time argument is used, 5 minutes before the system goes down the
  /run/nologin file is created to ensure that further logins shall not be
  allowed.
  
  ```
  
  > **Comments:**
  > **username_0**: *Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.* Wow, I had no idea `shutdown` could prevent logins. Manual testing indicates this might work. I'll test this out in CI to confirm.  
  > **username_0**: unfortunately the agent remains logged in if there is another pipeline already queued. So, the next pipeline starts, and is abruptly killed 5 minutes into the job. Is there a way to force each pipeline to use a new agent?  
  
  ---
  
  ### Answer by username_3 on 2022-06-03. Score: 0
  
  Some ideas to get you started:
  
  * Create a separate job to shut down the node and call that with a [build step](https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/) from your pipeline with "wait: false", e.g.: `build(job: "shutdown_host", wait: false)`. The "shutdown\_host" job would probably need to have a sleep statement or something before shutting down the node to avoid killing the node before the parent job has a chance to exit.
  * Execute a short shell script that forks & detaches a new process to shut down the host. Again you would probably need a sleep statement in the forked process to avoid shutting down the node before the Jenkins job exits. Also, in this case, you would need to disable the [Process Tree Killer](https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller) to avoid the forked process being killed.
  * Execute a step on the Jenkins master to ssh into your node to shut it down. But this would depend on the Jenkins master having ssh access to your node and administrative privileges to shut it down.
  
  I'm sure there are some other clever and creative ideas that I'm missing as well.
  
  > **Comments:**
  > **username_0**: Isn't there a race in all of these solutions? Suppose one job is executing, and a second is queued. If the second job starts while the first is sleeping in the background, then won't the second job be disrupted due to the reboot?  
  > **username_3**: Yes, there will be.  You can check in your pipeline code if there are any jobs running on a node (there should be other Stack Exchange questions that address how to do this) before rebooting, or you can use [resource locks](https://www.jenkins.io/doc/pipeline/steps/lockable-resources/) (which are kind of like semaphores) to prevent race conditions.  
  
  ---
  
  