Answers.Body: <p>I have tested it on my end with the following script:</p>
  <pre class="lang-java prettyprint-override"><code>#!/usr/bin/env nextflow
  
  reads_ch = Channel.fromFilePairs('/root/test_nextflow/*_{1,2}.fastq.gz', flat: true)
  
  
  process mapToRef {
    memory '1 GB'
    echo true
    cpus 1
  
    input:
    tuple val(sample_id), file(r1), file(r2) from reads_ch
  
    output:
    tuple val(sample_id), file(&quot;${sample_id}.sam&quot;) into sam_ch
  
    script:
    &quot;&quot;&quot;
    echo bwa mem <span class="math-container">${r1} {r2}
    if [ $</span>{sample_id} == &quot;ERR2512393&quot; ]; then sleep 10; fi
    touch ${sample_id}.sam
    &quot;&quot;&quot;
  }
  
  process samToBam {
    memory '1 GB'
    cpus 1
    echo true
  
    input:
    tuple val(sample_id), file(sam) from sam_ch
  
    output:
    tuple val(sample_id), file(&quot;${sample_id}.bam&quot;) into bam_ch
  
    script:
    &quot;&quot;&quot;
    echo samtools view -S -b <span class="math-container">${sample_id}.sam
    touch $</span>{sample_id}.bam
    &quot;&quot;&quot;
  }
  
  process sortBam {
    memory '1 GB'
    cpus 1
    input:
    tuple val(sample_id), file(bam) from bam_ch
  
    output:
    tuple val(sample_id), file(&quot;${sample_id}.sorted.bam&quot;) into sorted_bam_ch
  
    script:
    &quot;&quot;&quot;
    echo samtools sort <span class="math-container">${sample_id}.bam
    touch $</span>{sample_id}.sorted.bam
    &quot;&quot;&quot;
  }
  </code></pre>
  <p>It's producing the following result:</p>
  <p><a href="https://i.sstatic.net/w9Ng8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/w9Ng8.png" alt="enter image description here" /></a></p>
  <p>As you can see, the two processes are running in parallel (<code>2 of 2</code>)</p>
  <p>It could be that yours are limited by the resources? If your PC doesn't have 40GB of memory and 32 CPUs, it will run them sequentially.</p>
  <p><strong>EDIT</strong>: By adding <code>sleep 10</code> to <code>mapToRef</code>, we can clearly see both of them are running in parallel now:</p>
  <p><a href="https://i.sstatic.net/VaMTj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VaMTj.png" alt="enter image description here" /></a></p>
  <p><strong>EDIT 2</strong>: Added condition to &quot;sleep&quot; on one of the samples:</p>
  <p><a href="https://i.sstatic.net/YwBmX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YwBmX.png" alt="enter image description here" /></a></p>
  <p><strong>Final EDIT</strong>: be mindful of the resources you allocate to your processes. You could use <code>${task.cpus}</code> and <code>${task.memory.toGiga()}</code> to facilitate the configuration of your processes so that they are not different from what you have defined and what is actually used. See <a href="https://www.nextflow.io/docs/latest/process.html#cpus" rel="nofollow noreferrer">CPUs</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T19:47:46.493Z
Answers.Text: Thanks @Law.
  My execution screenshot also looks almost exactly the same.
  Can you replace `touch ...` on your script with `sleep 5` to emulate long running tasks and confirm if the processes are running parallelly? I have well over 20GB memory and 16 CPUs on this machine, so limited resource should not be an issue.
Answers.UserDisplayName: PPrasai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T19:49:16.237Z
Answers.Text: @PPrasai, check again my edit, i was doing it ;)
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T19:53:21.907Z
Answers.Text: When the first process shows `1 of 2`, does the second process start execution immediately? In my system, both reads from the first process need to be complete before the second process can start
Answers.UserDisplayName: PPrasai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T19:58:31.187Z
Answers.Text: Check my second edit. I have added a condition to `mapToRef` to "sleep" if one of the `sample_id` match one of the samples. As you can see, both of them runs in parallel first, since one of the samples doesn't match the condition, the other processes are executed afterward.
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:02:02.547Z
Answers.Text: I seems to be working on your end...
  Can you spot any issues with my code?
Answers.UserDisplayName: PPrasai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:04:10.363Z
Answers.Text: Try to lower the ressources specified. Also, in your `Input` channels, instead of specifying, for example, `tuple val(sample_id), file("${sample_id}.sam") from sam_ch`, you should rather do `tuple val(sample_id), file(sam) from sam_ch` and in the script, do `~/biotools/SAMTOOLS/samtools-1.12/samtools view -S -b ${sam} > ${sample_id}.bam`
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:06:09.393Z
Answers.Text: Also, I just noticed something... why is there 60 CPU defined in your `mapToRef` when you gave it 16 CPUs ?
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:10:45.583Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/129268/discussion-between-law-and-pprasai).
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:11:02.343Z
Answers.Text: Thanks @Law!
  Reducing `cpus` to 4 on process and `mapToRef`, and `memory` to 4 GB make it all work parallelly! Thank you so much. Could you update your answer so that I can approve it
Answers.UserDisplayName: PPrasai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T20:12:32.233Z
Answers.Text: Well, I did say toward the end the following `It could be that your are limited by the ressources? If you PC doesn't have 40GB of memory and 32 CPUs, he will run them sequentially.` Also, you're welcome.
Answers.UserDisplayName: user324810
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T19:42:41.057Z
Answers.LastActivityDate: 2022-02-12T13:48:15.850Z
Answers.LastEditDate: 2022-02-12T13:48:15.850Z
Answers.UserDisplayName: user324810
Body: <p>I have written a Nextflow script with three process:</p>
  <ul>
  <li>The first process takes a pair of <code>fastq</code> files and aligns with reference genome. The process writes the resulting <code>SAM</code> file into <code>sam channel</code>.</li>
  <li>Second process takes input from the <code>sam channel</code> and creates a <code>BAM</code> file from it, and writes it into <code>bam channel</code>.</li>
  <li>Final process reads from <code>bam channel</code> and sorts the <code>BAM</code> files.</li>
  </ul>
  <p>Following is the entire code:</p>
  <pre><code>#!~/bin nextflow
  
  reads_ch = Channel.fromFilePairs('raw/*_{1,2}.fastq.gz', flat: true)
  ref_genome = file('reference/genome/index/ref_genome')
  
  process mapToRef {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file(read1), file(read2) from reads_ch
      
      output:
      tuple val(sample_id), file(&quot;${sample_id}.sam&quot;) into sam_ch
  
      script:
      &quot;&quot;&quot;
      ~/biotools/bwa/bwa mem -t 60 <span class="math-container">${ref_genome} $</span>{read1} <span class="math-container">${read2} &gt; $</span>{sample_id}.sam
      &quot;&quot;&quot;   
  }
  
  process samToBam {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file(&quot;${sample_id}.sam&quot;) from sam_ch
  
      output:
      tuple val(sample_id), file(&quot;${sample_id}.bam&quot;) into bam_ch
  
      script:
      &quot;&quot;&quot;
      ~/biotools/SAMTOOLS/samtools-1.12/samtools view -S -b <span class="math-container">${sample_id}.sam &gt; $</span>{sample_id}.bam 
      &quot;&quot;&quot;
  }
  
  process sortBam {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file(&quot;${sample_id}.bam&quot;) from bam_ch
  
      output:
      tuple val(sample_id), file(&quot;${sample_id}.sorted.bam&quot;) into sorted_bam_ch
  
      script:
      &quot;&quot;&quot;
      ~/biotools/SAMTOOLS/samtools-1.12/samtools sort <span class="math-container">${sample_id}.bam -o $</span>{sample_id}.sorted.bam
      &quot;&quot;&quot;
  }
  </code></pre>
  <p>All three processes successfully execute their commands. However, they are executed sequentially. For example, the first process <code>mapToRef</code> runs <strong>sequentially</strong> for all read pairs, followed by <code>samToBam</code>, which too, runs sequentially after all <code>mapToRef</code>s have finished execution. I was under impression that parallelization would &quot;work&quot; out of the box, but I must be doing something wrong. I am a complete beginner in Nextflow and would greatly appreciate any help regarding this.</p>
  <p><strong>EDIT</strong></p>
  <p>Output of <code>reads_ch.view()</code>:</p>
  <pre><code>[ERR2512393, /home/work/raw/ERR2512393_1.fastq.gz, /home/work/raw/ERR2512393_2.fastq.gz]
  [ERR2512394, /home/work/raw/ERR2512394_1.fastq.gz, /home/work/raw/ERR2512394_2.fastq.gz]
  [ERR2512391, /home/work/raw/ERR2512391_1.fastq.gz, /home/work/raw/ERR2512391_2.fastq.gz]
  [ERR2512392, /home/work/raw/ERR2512392_1.fastq.gz, /home/work/raw/ERR2512392_2.fastq.gz]
  [ERR2512390, /home/work/raw/ERR2512390_1.fastq.gz, /home/work/raw/ERR2512390_2.fastq.gz]
  </code></pre>
  
Tags: <nextflow>
Title: Why are my Nextflow processes not executing in parallel?
ThreadText: # Why are my Nextflow processes not executing in parallel?
  
  (Asked by: username_0 on 2021-09-04)
  
  I have written a Nextflow script with three process:
  
  * The first process takes a pair of `fastq` files and aligns with reference genome. The process writes the resulting `SAM` file into `sam channel`.
  * Second process takes input from the `sam channel` and creates a `BAM` file from it, and writes it into `bam channel`.
  * Final process reads from `bam channel` and sorts the `BAM` files.
  
  Following is the entire code:
  
  ```
  #!~/bin nextflow
  
  reads_ch = Channel.fromFilePairs('raw/*_{1,2}.fastq.gz', flat: true)
  ref_genome = file('reference/genome/index/ref_genome')
  
  process mapToRef {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file(read1), file(read2) from reads_ch
      
      output:
      tuple val(sample_id), file("${sample_id}.sam") into sam_ch
  
      script:
      """
      ~/biotools/bwa/bwa mem -t 60 ${ref_genome} ${read1} ${read2} > ${sample_id}.sam
      """   
  }
  
  process samToBam {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file("${sample_id}.sam") from sam_ch
  
      output:
      tuple val(sample_id), file("${sample_id}.bam") into bam_ch
  
      script:
      """
      ~/biotools/SAMTOOLS/samtools-1.12/samtools view -S -b ${sample_id}.sam > ${sample_id}.bam 
      """
  }
  
  process sortBam {
      memory '20 GB'
      cpus 16
      input:
      tuple val(sample_id), file("${sample_id}.bam") from bam_ch
  
      output:
      tuple val(sample_id), file("${sample_id}.sorted.bam") into sorted_bam_ch
  
      script:
      """
      ~/biotools/SAMTOOLS/samtools-1.12/samtools sort ${sample_id}.bam -o ${sample_id}.sorted.bam
      """
  }
  
  ```
  
  All three processes successfully execute their commands. However, they are executed sequentially. For example, the first process `mapToRef` runs **sequentially** for all read pairs, followed by `samToBam`, which too, runs sequentially after all `mapToRef`s have finished execution. I was under impression that parallelization would "work" out of the box, but I must be doing something wrong. I am a complete beginner in Nextflow and would greatly appreciate any help regarding this.
  
  **EDIT**
  
  Output of `reads_ch.view()`:
  
  ```
  [ERR2512393, /home/work/raw/ERR2512393_1.fastq.gz, /home/work/raw/ERR2512393_2.fastq.gz]
  [ERR2512394, /home/work/raw/ERR2512394_1.fastq.gz, /home/work/raw/ERR2512394_2.fastq.gz]
  [ERR2512391, /home/work/raw/ERR2512391_1.fastq.gz, /home/work/raw/ERR2512391_2.fastq.gz]
  [ERR2512392, /home/work/raw/ERR2512392_1.fastq.gz, /home/work/raw/ERR2512392_2.fastq.gz]
  [ERR2512390, /home/work/raw/ERR2512390_1.fastq.gz, /home/work/raw/ERR2512390_2.fastq.gz]
  
  ```
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-04. Score: 5 | Accepted answer
  
  I have tested it on my end with the following script:
  
  ```java
  #!/usr/bin/env nextflow
  
  reads_ch = Channel.fromFilePairs('/root/test_nextflow/*_{1,2}.fastq.gz', flat: true)
  
  process mapToRef {
    memory '1 GB'
    echo true
    cpus 1
  
    input:
    tuple val(sample_id), file(r1), file(r2) from reads_ch
  
    output:
    tuple val(sample_id), file("${sample_id}.sam") into sam_ch
  
    script:
    """
    echo bwa mem ${r1} {r2}
    if [ ${sample_id} == "ERR2512393" ]; then sleep 10; fi
    touch ${sample_id}.sam
    """
  }
  
  process samToBam {
    memory '1 GB'
    cpus 1
    echo true
  
    input:
    tuple val(sample_id), file(sam) from sam_ch
  
    output:
    tuple val(sample_id), file("${sample_id}.bam") into bam_ch
  
    script:
    """
    echo samtools view -S -b ${sample_id}.sam
    touch ${sample_id}.bam
    """
  }
  
  process sortBam {
    memory '1 GB'
    cpus 1
    input:
    tuple val(sample_id), file(bam) from bam_ch
  
    output:
    tuple val(sample_id), file("${sample_id}.sorted.bam") into sorted_bam_ch
  
    script:
    """
    echo samtools sort ${sample_id}.bam
    touch ${sample_id}.sorted.bam
    """
  }
  
  ```
  
  It's producing the following result:
  
  [![enter image description here](https://i.sstatic.net/w9Ng8.png)](https://i.sstatic.net/w9Ng8.png)
  
  As you can see, the two processes are running in parallel (`2 of 2`)
  
  It could be that yours are limited by the resources? If your PC doesn't have 40GB of memory and 32 CPUs, it will run them sequentially.
  
  **EDIT**: By adding `sleep 10` to `mapToRef`, we can clearly see both of them are running in parallel now:
  
  [![enter image description here](https://i.sstatic.net/VaMTj.png)](https://i.sstatic.net/VaMTj.png)
  
  **EDIT 2**: Added condition to "sleep" on one of the samples:
  
  [![enter image description here](https://i.sstatic.net/YwBmX.png)](https://i.sstatic.net/YwBmX.png)
  
  **Final EDIT**: be mindful of the resources you allocate to your processes. You could use `${task.cpus}` and `${task.memory.toGiga()}` to facilitate the configuration of your processes so that they are not different from what you have defined and what is actually used. See [CPUs](https://www.nextflow.io/docs/latest/process.html#cpus)
  
  > **Comments:**
  > **username_0**: Thanks @Law. My execution screenshot also looks almost exactly the same. Can you replace `touch ...` on your script with `sleep 5` to emulate long running tasks and confirm if the processes are running parallelly? I have well over 20GB memory and 16 CPUs on this machine, so limited resource should not be an issue.  
  > **username_1**: @username_0, check again my edit, i was doing it ;)  
  > **username_0**: When the first process shows `1 of 2`, does the second process start execution immediately? In my system, both reads from the first process need to be complete before the second process can start  
  > **username_1**: Check my second edit. I have added a condition to `mapToRef` to "sleep" if one of the `sample_id` match one of the samples. As you can see, both of them runs in parallel first, since one of the samples doesn't match the condition, the other processes are executed afterward.  
  > **username_0**: I seems to be working on your end... Can you spot any issues with my code?  
  > **username_1**: Try to lower the ressources specified. Also, in your `Input` channels, instead of specifying, for example, `tuple val(sample_id), file("${sample_id}.sam") from sam_ch`, you should rather do `tuple val(sample_id), file(sam) from sam_ch` and in the script, do `~/biotools/SAMTOOLS/samtools-1.12/samtools view -S -b ${sam} > ${sample_id}.bam`  
  > **username_1**: Also, I just noticed something... why is there 60 CPU defined in your `mapToRef` when you gave it 16 CPUs ?  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/129268/discussion-between-law-and-username_0).  
  > **username_0**: Thanks @Law! Reducing `cpus` to 4 on process and `mapToRef`, and `memory` to 4 GB make it all work parallelly! Thank you so much. Could you update your answer so that I can approve it  
  > **username_1**: Well, I did say toward the end the following `It could be that your are limited by the ressources? If you PC doesn't have 40GB of memory and 32 CPUs, he will run them sequentially.` Also, you're welcome.  
  
  ---
  
  