Answers.Body: <p>How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the <a href="https://www.nextflow.io/docs/latest/metadata.html#completion-handler" rel="nofollow noreferrer"><code>workflow.onComplete</code></a> and <a href="https://www.nextflow.io/docs/latest/metadata.html#error-handler" rel="nofollow noreferrer"><code>workflow.onError</code></a> event handlers. Consider resuming your workflow with something like the following appended to your script:</p>
  <pre><code>workflow.onComplete {
  
    log.info &quot;Workflow completed at: <span class="math-container">${workflow.complete}"
    log.info "Time taken: $</span>{workflow.duration}&quot;
    log.info &quot;Execution status: ${workflow.success ? 'success' : 'failed'}&quot;
  }
  
  workflow.onError {
  
    log.info &quot;Execution halted: ${workflow.errorMessage}&quot;
  }
  </code></pre>
  <p>Note that the workflow event handlers may also be defined <a href="https://www.nextflow.io/docs/latest/metadata.html#decoupling-metadata" rel="nofollow noreferrer">inside</a> your <code>nextflow.config</code> if preferred.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-28T14:45:01.337Z
Answers.Text: Thanks, @Steve. I added an example to my post. Seems that one way or another, the example I show above should have completed because `Nextflow` has nothing left to do.
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T00:14:05.823Z
Answers.Text: @MarkEbbert It's hard to say what the problem is here exactly without the nextflow.log or a reproducible example. But just because all of the tasks have completed (regardless of the exit status) doesn't necessarily mean Nextflow has nothing left to do. For example, the publishDir directive works asynchronously and can be used (as you know) to 'copy' task outputs. If these are large and there are many of these, Nextflow will hang around copying files from the queue until well after the last task has completed.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T19:07:17.953Z
Answers.Text: Thanks, @Steve. I just added part of the `.nextflow.log` file. Looks like `Nextflow` thinks it's hung, too? I haven't tested whether this is reproducible, by this is the code for my related question. I included a link to it above. This one may just remain a mystery.
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T19:25:06.070Z
Answers.Text: I can't edit my above comment, but should read "..., but this is from the code..."
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-02T02:02:32.290Z
Answers.Text: @MarkEbbert Correct - this looks like a bug. It might have something to do with the errorStrategy set to 'ignore' for all processes, potentially. Basically, you need to hit [this condition](https://github.com/nextflow-io/nextflow/blob/v21.10.6/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy#L412) to break out of the poll loop. From the partial log you've posted, it looks like the runningQueue is empty, but I don't know about the pendingQueue.size() or if session.isTerminated().
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-02T02:02:56.337Z
Answers.Text: @MarkEbbert You might like to open an [issue](https://github.com/nextflow-io/nextflow/issues) with the full log attached.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-02T20:57:08.813Z
Answers.Text: Thanks, @Steve. Submitted: https://github.com/nextflow-io/nextflow/issues/2693. I'll report back if I learn something.
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-27T14:00:35.317Z
Answers.LastActivityDate: 2022-02-27T14:00:35.317Z
Answers.UserDisplayName: Steve
Body: <h2>Bug fixed</h2>
  <p>This issue appears to have been fixed in latest <code>Nextflow</code>: <a href="https://github.com/nextflow-io/nextflow/issues/2693" rel="nofollow noreferrer">https://github.com/nextflow-io/nextflow/issues/2693</a></p>
  <hr />
  <p>I've been using <code>Nextflow</code> for various workflows on a cluster using the <code>SLURM</code> scheduler. I've noticed that the 'parent' job does not shut down when the <code>Nextflow</code> workflow finishes successfully. I can cancel the job, but then <code>Nextflow</code> doesn't get a chance to clean things up and also doesn't write out the <code>trace</code>, <code>report</code>, etc. The parent job does end properly if the workflow ends with an error.</p>
  <p><strong>Question</strong>: Why doesn't the parent <code>Nextflow</code> job end upon successful completion?</p>
  <p>I saw <a href="https://github.com/nextflow-io/nextflow/issues/968" rel="nofollow noreferrer">a post on the <code>Nextflow</code> GitHub</a> where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that <code>Nextflow</code> should handle.</p>
  <p>Here's my <code>SLURM</code> submission script that I submit using <code>sbatch run_workflow.slurm</code></p>
  <p><strong>run_workflow.slurm</strong></p>
  <pre><code>#!/bin/bash
  #SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
  #SBATCH --job-name=NF_Parent_Identify_D&amp;C_regions   # Job name
  #SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
  #SBATCH --mem=5G                                    # Total memory requested
  #SBATCH --partition=&lt;partition&gt;                     # Partition/queue to run the job in. (REQUIRED
  #SBATCH -e slurm-%j.err                             # Error file for this job.
  #SBATCH -o slurm-%j.out                             # Output file for this job.
  #SBATCH -A &lt;account&gt;                                # Project allocation account name (REQUIRED)
  
  module load ccs/java/jdk1.8.0_202
  nextflow run identify_dark_and_camouflaged_regions.nf \
      -with-report \
      -with-trace \
      -with-dag idcr-workflow.png \
      -with-timeline \
      -resume
  </code></pre>
  <h1>Update 1</h1>
  <p>I'm adding example <code>Nextflow</code> output from a different workflow (I don't have the <code>SLURM</code> output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added <code>errorStrategy 'ignore'</code> to all of the processes listed:</p>
  <pre><code>executor &gt;  slurm (876)
  [06/c5672b] process &gt; samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  [45/26a046] process &gt; Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
  [b5/0f2d7c] process &gt; sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  </code></pre>
  <p>So, regardless of whether the workflow completed <code>successfully</code>, <code>Nextflow</code> has nothing left to do, but it's still running.</p>
  <h1>Update 2 (adding partial <code>.nextflow.log</code> file)</h1>
  <p>Per @Steve's point, I checked the <code>.nextflow.log</code> file and am including part of the file, here. In this case, I don't think <code>Nextflow</code> has any tasks related to <code>publishDir</code> because I'm not using that in this workflow. The job that's still running is from the <code>Nextflow</code> workflow in my GitHub repository for my answer posted in my related question <a href="https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c">Samtools sort: most efficient memory and thread settings for many samples on a cluster</a>. Whether it's reproducible, I haven't checked. Not sure how common this issue is.</p>
  <p>In either case, my interpretation of the <code>.nextflow.log</code> file below is that <code>Nextflow</code> thinks it's hung, too. I just don't know why, or how to exit cleanly.</p>
  <p><strong>Partial view of <code>.nextflow.log</code></strong></p>
  <pre><code>...
  
  Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm &gt; tasks to be completed: 1 -- submitted tasks are shown below
  ~&gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
  Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed &gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
  Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  
  &lt; above line repeats hundreds of times &gt;
  
  ...
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-17T05:21:49.237Z
Comments.Text: Can you please create an answer with your accepted solution, so that the StackExchange algorithms don't confuse this with an unanswered question?
Comments.UserDisplayName: gringer
Tags: <nextflow><slurm>
Title: Nextflow scheduler job doesn't end upon successful workflow completion
ThreadText: # Nextflow scheduler job doesn't end upon successful workflow completion
  
  (Asked by: username_0 on 2022-02-26)
  
  ## Bug fixed
  
  This issue appears to have been fixed in latest `Nextflow`: <https://github.com/nextflow-io/nextflow/issues/2693>
  
  ---
  
  I've been using `Nextflow` for various workflows on a cluster using the `SLURM` scheduler. I've noticed that the 'parent' job does not shut down when the `Nextflow` workflow finishes successfully. I can cancel the job, but then `Nextflow` doesn't get a chance to clean things up and also doesn't write out the `trace`, `report`, etc. The parent job does end properly if the workflow ends with an error.
  
  **Question**: Why doesn't the parent `Nextflow` job end upon successful completion?
  
  I saw [a post on the `Nextflow` GitHub](https://github.com/nextflow-io/nextflow/issues/968) where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that `Nextflow` should handle.
  
  Here's my `SLURM` submission script that I submit using `sbatch run_workflow.slurm`
  
  **run\_workflow.slurm**
  
  ```
  #!/bin/bash
  #SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
  #SBATCH --job-name=NF_Parent_Identify_D&C_regions   # Job name
  #SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
  #SBATCH --mem=5G                                    # Total memory requested
  #SBATCH --partition=<partition>                     # Partition/queue to run the job in. (REQUIRED
  #SBATCH -e slurm-%j.err                             # Error file for this job.
  #SBATCH -o slurm-%j.out                             # Output file for this job.
  #SBATCH -A <account>                                # Project allocation account name (REQUIRED)
  
  module load ccs/java/jdk1.8.0_202
  nextflow run identify_dark_and_camouflaged_regions.nf \
      -with-report \
      -with-trace \
      -with-dag idcr-workflow.png \
      -with-timeline \
      -resume
  
  ```
  
  # Update 1
  
  I'm adding example `Nextflow` output from a different workflow (I don't have the `SLURM` output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added `errorStrategy 'ignore'` to all of the processes listed:
  
  ```
  executor >  slurm (876)
  [06/c5672b] process > samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  [45/26a046] process > Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
  [b5/0f2d7c] process > sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...
  
  ```
  
  So, regardless of whether the workflow completed `successfully`, `Nextflow` has nothing left to do, but it's still running.
  
  # Update 2 (adding partial `.nextflow.log` file)
  
  Per @username_2's point, I checked the `.nextflow.log` file and am including part of the file, here. In this case, I don't think `Nextflow` has any tasks related to `publishDir` because I'm not using that in this workflow. The job that's still running is from the `Nextflow` workflow in my GitHub repository for my answer posted in my related question [Samtools sort: most efficient memory and thread settings for many samples on a cluster](https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c). Whether it's reproducible, I haven't checked. Not sure how common this issue is.
  
  In either case, my interpretation of the `.nextflow.log` file below is that `Nextflow` thinks it's hung, too. I just don't know why, or how to exit cleanly.
  
  **Partial view of `.nextflow.log`**
  
  ```
  ...
  
  Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
  ~> TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
  Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
  Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
  
  < above line repeats hundreds of times >
  
  ...
  
  ```
  
  **Tags:** nextflow, slurm
  
  > **Comments:**
  > **username_1**: Can you please create an answer with your accepted solution, so that the StackExchange algorithms don't confuse this with an unanswered question?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-27. Score: 2
  
  How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the [`workflow.onComplete`](https://www.nextflow.io/docs/latest/metadata.html#completion-handler) and [`workflow.onError`](https://www.nextflow.io/docs/latest/metadata.html#error-handler) event handlers. Consider resuming your workflow with something like the following appended to your script:
  
  ```
  workflow.onComplete {
  
    log.info "Workflow completed at: ${workflow.complete}"
    log.info "Time taken: ${workflow.duration}"
    log.info "Execution status: ${workflow.success ? 'success' : 'failed'}"
  }
  
  workflow.onError {
  
    log.info "Execution halted: ${workflow.errorMessage}"
  }
  
  ```
  
  Note that the workflow event handlers may also be defined [inside](https://www.nextflow.io/docs/latest/metadata.html#decoupling-metadata) your `nextflow.config` if preferred.
  
  > **Comments:**
  > **username_0**: Thanks, @username_2. I added an example to my post. Seems that one way or another, the example I show above should have completed because `Nextflow` has nothing left to do.  
  > **username_2**: @username_0 It's hard to say what the problem is here exactly without the nextflow.log or a reproducible example. But just because all of the tasks have completed (regardless of the exit status) doesn't necessarily mean Nextflow has nothing left to do. For example, the publishDir directive works asynchronously and can be used (as you know) to 'copy' task outputs. If these are large and there are many of these, Nextflow will hang around copying files from the queue until well after the last task has completed.  
  > **username_0**: Thanks, @username_2. I just added part of the `.nextflow.log` file. Looks like `Nextflow` thinks it's hung, too? I haven't tested whether this is reproducible, by this is the code for my related question. I included a link to it above. This one may just remain a mystery.  
  > **username_0**: I can't edit my above comment, but should read "..., but this is from the code..."  
  > **username_2**: @username_0 Correct - this looks like a bug. It might have something to do with the errorStrategy set to 'ignore' for all processes, potentially. Basically, you need to hit [this condition](https://github.com/nextflow-io/nextflow/blob/v21.10.6/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy#L412) to break out of the poll loop. From the partial log you've posted, it looks like the runningQueue is empty, but I don't know about the pendingQueue.size() or if session.isTerminated().  
  > **username_2**: @username_0 You might like to open an [issue](https://github.com/nextflow-io/nextflow/issues) with the full log attached.  
  > **username_0**: Thanks, @username_2. Submitted: https://github.com/nextflow-io/nextflow/issues/2693. I'll report back if I learn something.  
  
  ---
  
  