Answers.Body: <p>This is a result of nextflow channel types, <a href="https://www.nextflow.io/docs/latest/channel.html#channel-types" rel="nofollow noreferrer">queue- and value-channels</a>. Nextflow will make fifo combinations of your channels to run <code>bwa_align</code> until either or both channel are empty. <code>index_ch</code> contains only one item, so once it is used there are no more valid combinations to start new  <code>bwa_align</code> processes.</p>
  <p>Either <a href="https://www.nextflow.io/docs/latest/channel.html#value" rel="nofollow noreferrer">create</a> your <code>index_ch</code> as value channel, or keep the index out of a channel entirely and refer to it as a <a href="https://www.nextflow.io/docs/latest/config.html#scope-params" rel="nofollow noreferrer">parameter</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-14T04:13:10.000Z
Answers.Text: Thanks for your response! I did as you suggested and used `Channel.value(params...)` for both my index and ref channels. While this did work I still seem to be missing something. While my `bwa_align` process is now aligning both sets of files, the `mapping_stats` still is 1 of 1. I was under the impression that any output from a process is a value channel, so this should work. Even more strange is this process only takes a single input. Do you have any advice for how to rectify this or what I am misunderstanding?
Answers.UserDisplayName: bhumm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-14T12:54:59.257Z
Answers.Text: Afaik a process will only emit a value channel if ALL its input channels were value channels, otherwise it will be a queue chanel. Make a new question if you have a new question, it is hard to debug your new issue without the exact code. The output you show seems to suggest you are using the `-resume` option, maybe this is messing with the correct execution after you changed the flow.
Answers.UserDisplayName: Pallie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-14T14:09:00.023Z
Answers.Text: Apologies, I was too impatient and responded prematurely. The executor updated to 2 of 2 for the `mapping_stats` following the completion of the second `bwa_align` process. Looks like everything is working as expected. Thanks again for your help!
Answers.UserDisplayName: bhumm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-13T08:56:39.110Z
Answers.LastActivityDate: 2024-02-13T08:56:39.110Z
Answers.UserDisplayName: Pallie
Body: <p>I have written a pipeline that is composed of three processes. I seem to be misunderstanding how Nextflow passes the outputs from one process to the next or creates a queue of files. I am using a tumor and paired normal sample from SRA (<a href="https://www.ncbi.nlm.nih.gov/sra/SRX21185781%5Baccn%5D" rel="nofollow noreferrer">SRX21185781</a>) to prototype this pipeline.</p>
  <p>My first process, <code>trim_QC</code> works fine and outputs the expected number of trimmed fastqs. However, the next process, <code>bwa_align</code> only creates a single bam from one of the two output from the preceding process. My terminal looks like this:</p>
  <pre><code>N E X T F L O W  ~  version 23.10.1
  Launching `../main.nf` [sick_church] DSL2 - revision: beb03cc0c6
  
  [6e/48d4f6] process &gt; trim_QC (SRR25452858_normal)                  [100%] 2 of 2, cached: 2 ✔
  [39/1b8cb2] process &gt; bwa_align (SRR25452858_normal)                [100%] 1 of 1, cached: 1 ✔
  [5f/e598c1] process &gt; mapping_stats (SRR25452858_normal_sorted.bam) [100%] 1 of 1, cached: 1 ✔
  ...
  </code></pre>
  <p>and when running with <code>-with-trace</code> it is even more obvious only one bam file is being processed by the downstream processes:</p>
  <pre><code>task_id hash    native_id   name    status  exit    submit  duration    realtime    %cpu    peak_rss    peak_vmem   rchar   wchar
  2   d3/0bd651   71127   trim_QC (SRR25452863_tumor) CACHED  0   2024-02-07 16:28:37.449 1h 21m 10s  1h 21m 10s  -   -   -   -   -
  1   6e/48d4f6   71128   trim_QC (SRR25452858_normal)    CACHED  0   2024-02-07 16:28:37.459 1h 16m 32s  1h 16m 32s  -   -   -   -   -
  3   39/1b8cb2   2754    bwa_align (SRR25452858_normal)  CACHED  0   2024-02-08 10:08:46.033 4h 18m 10s  4h 18m 10s  -   -   -   -   -
  4   5f/e598c1   9282    mapping_stats (SRR25452858_normal_sorted.bam)   CACHED  0   2024-02-08 14:26:55.823 53.9s   53.8s   -   -   -   -   -
  
  </code></pre>
  <p>I am expecting for the <code>bwa-align</code> and <code>mapping_stats</code> processes to also be n of 2.</p>
  <p>I have tried explicitly designating the <code>trim_QC.out</code> as a channel (<code>Channel.of(trim_QC.out)</code>) or using an operator to return a queue channel object (i.e., <code>.collect</code>, <code>.collectFile()</code>, etc.) but this either returns errors or fails.</p>
  <p>If I try to visualize the outputs from the <code>trim_QC</code> process by adding this to my workflow: <code>trim_QC.out.trimmed_reads.view{it}</code>, it does show both file pairs in my terminal, but only the first one is processed by <code>bwa_align</code> and subsequent steps. Like so:</p>
  <pre><code>...
  [6e/48d4f6] process &gt; trim_QC (SRR25452858_normal)            [100%] 2 of 2, cached: 2 ✔
  [39/1b8cb2] process &gt; bwa_align (SRR25452858_normal)          [100%] 1 of 1, cached: 1 ✔
  [5f/e598c1] process &gt; mapping_stats (SRR25452858_normal_so... [100%] 1 of 1, cached: 1 ✔
  hello
  
  [SRR25452858_normal, /Users/me/Desktop/fastq2/work/6e/48d4f61cae83ff903abceaadf715e5/SRR25452858_normal_1_val_1.fq.gz, /Users/me/Desktop/fastq2/work/6e/48d4f61cae83ff903abceaadf715e5/SRR25452858_normal_2_val_2.fq.gz]
  Analysis complete for SRR25452858_normal_1_val_1.fq.gz
  Analysis complete for SRR25452858_normal_2_val_2.fq.gz
  
  [SRR25452863_tumor, /Users/me/Desktop/fastq2/work/d3/0bd6515fcedfc060596b75dd90c28a/SRR25452863_tumor_1_val_1.fq.gz, /Users/me/Desktop/fastq2/work/d3/0bd6515fcedfc060596b75dd90c28a/SRR25452863_tumor_2_val_2.fq.gz]
  Analysis complete for SRR25452863_tumor_1_val_1.fq.gz
  Analysis complete for SRR25452863_tumor_2_val_2.fq.gz
  </code></pre>
  <p>I am unsure if I am misunderstanding what Nextflow is capable of or how channels work. What I am trying to accomplish is trim adapters and QC result, align trimmed reads followed by indexing, and finally calculate mappings statistics. Any advice is appreciated.</p>
  <p>Here is my entire script:</p>
  <pre><code>#!/usr/bin/env nextflow
  nextflow.enable.dsl = 2
  
  params.fastq = &quot;<span class="math-container">$projectDir/fastq2/*_{1,2}*"
  params.index_dir = "$</span>projectDir/RefGenome/BWAIndex&quot;
  params.ref = &quot;hg38.fa&quot;
  params.out_dir = &quot;$projectDir/Results&quot;
  
  workflow {
  
      fastq_ch = Channel.fromFilePairs(params.fastq)
      index_ch = Channel.fromPath(params.index_dir)
      ref_ch = Channel.of(params.ref)
  
      trim_QC(fastq_ch)
      bwa_align(index_ch, ref_ch, trim_QC.out.trimmed_reads)
      mapping_stats(bwa_align.out.sorted_bam)
  }
  
  process trim_QC {
      debug true
      tag &quot;$sample_id&quot;
  
      publishDir = [
                      [
                          path: { &quot;${params.out_dir}/trimmed/&quot; },
                          mode: 'copy',
                          overwrite: false
                      ],
  
                      [
                          path: { &quot;${params.out_dir}/stats/&quot; },
                          mode: 'copy',
                          pattern: &quot;*fastqc*&quot;,
                          overwrite: false
                      ]
                  ]
  
      input:
      tuple val(sample_id), path(fastq)
  
      output:
      tuple val(&quot;${sample_id}&quot;), path(&quot;*_val_1.fq.gz&quot;), path(&quot;*_val_2.fq.gz&quot;), emit: trimmed_reads
      path(&quot;*fastqc*&quot;)
  
      script:
      def (read1, read2) = fastq
      &quot;&quot;&quot;
      trim_galore --paired --fastqc <span class="math-container">${read1} $</span>{read2}
      &quot;&quot;&quot;
  }
  
  process bwa_align {
      debug true
      tag &quot;$sample_id&quot;
  
      publishDir &quot;${params.out_dir}/bwa_aligned/&quot;, mode: 'copy', overwrite: false
  
      input:
      path index_dir
      val ref
      tuple val(sample_id), path(read1), path(read2)
  
      output:
      path(&quot;<span class="math-container">${sample_id}_sorted.bam"), emit: sorted_bam
  path("$</span>{sample_id}_sorted.bam.bai&quot;), emit: bam_index
  
      script:
      &quot;&quot;&quot;
      bwa mem -t 2 <span class="math-container">${index_dir}/$</span>{ref} <span class="math-container">${read1} $</span>{read2} | samtools sort -@2 -o &quot;<span class="math-container">${sample_id}_sorted.bam" -
  samtools index $</span>{sample_id}_sorted.bam &gt; &quot;${sample_id}_sorted.bam.bai&quot;
      &quot;&quot;&quot;
  }
  
  process mapping_stats {
      debug true
      tag &quot;$sorted_bam&quot;
  
      publishDir &quot;${params.out_dir}/stats/&quot;, mode: 'copy', overwrite: false
  
      input:
      path(sorted_bam)
  
      output:
      path(&quot;${sorted_bam}_mapping_stats.txt&quot;)
  
      script:
      &quot;&quot;&quot;
      samtools flagstat <span class="math-container">${sorted_bam} &gt; "$</span>{sorted_bam}_mapping_stats.txt&quot;
      &quot;&quot;&quot;
  }
  
  </code></pre>
  
Tags: <nextflow>
Title: Nextflow: Properly chaining process outputs
ThreadText: # Nextflow: Properly chaining process outputs
  
  (Asked by: username_0 on 2024-02-12)
  
  I have written a pipeline that is composed of three processes. I seem to be misunderstanding how Nextflow passes the outputs from one process to the next or creates a queue of files. I am using a tumor and paired normal sample from SRA ([SRX21185781](https://www.ncbi.nlm.nih.gov/sra/SRX21185781%5Baccn%5D)) to prototype this pipeline.
  
  My first process, `trim_QC` works fine and outputs the expected number of trimmed fastqs. However, the next process, `bwa_align` only creates a single bam from one of the two output from the preceding process. My terminal looks like this:
  
  ```
  N E X T F L O W  ~  version 23.10.1
  Launching `../main.nf` [sick_church] DSL2 - revision: beb03cc0c6
  
  [6e/48d4f6] process > trim_QC (SRR25452858_normal)                  [100%] 2 of 2, cached: 2 ✔
  [39/1b8cb2] process > bwa_align (SRR25452858_normal)                [100%] 1 of 1, cached: 1 ✔
  [5f/e598c1] process > mapping_stats (SRR25452858_normal_sorted.bam) [100%] 1 of 1, cached: 1 ✔
  ...
  
  ```
  
  and when running with `-with-trace` it is even more obvious only one bam file is being processed by the downstream processes:
  
  ```
  task_id hash    native_id   name    status  exit    submit  duration    realtime    %cpu    peak_rss    peak_vmem   rchar   wchar
  2   d3/0bd651   71127   trim_QC (SRR25452863_tumor) CACHED  0   2024-02-07 16:28:37.449 1h 21m 10s  1h 21m 10s  -   -   -   -   -
  1   6e/48d4f6   71128   trim_QC (SRR25452858_normal)    CACHED  0   2024-02-07 16:28:37.459 1h 16m 32s  1h 16m 32s  -   -   -   -   -
  3   39/1b8cb2   2754    bwa_align (SRR25452858_normal)  CACHED  0   2024-02-08 10:08:46.033 4h 18m 10s  4h 18m 10s  -   -   -   -   -
  4   5f/e598c1   9282    mapping_stats (SRR25452858_normal_sorted.bam)   CACHED  0   2024-02-08 14:26:55.823 53.9s   53.8s   -   -   -   -   -
  
  ```
  
  I am expecting for the `bwa-align` and `mapping_stats` processes to also be n of 2.
  
  I have tried explicitly designating the `trim_QC.out` as a channel (`Channel.of(trim_QC.out)`) or using an operator to return a queue channel object (i.e., `.collect`, `.collectFile()`, etc.) but this either returns errors or fails.
  
  If I try to visualize the outputs from the `trim_QC` process by adding this to my workflow: `trim_QC.out.trimmed_reads.view{it}`, it does show both file pairs in my terminal, but only the first one is processed by `bwa_align` and subsequent steps. Like so:
  
  ```
  ...
  [6e/48d4f6] process > trim_QC (SRR25452858_normal)            [100%] 2 of 2, cached: 2 ✔
  [39/1b8cb2] process > bwa_align (SRR25452858_normal)          [100%] 1 of 1, cached: 1 ✔
  [5f/e598c1] process > mapping_stats (SRR25452858_normal_so... [100%] 1 of 1, cached: 1 ✔
  hello
  
  [SRR25452858_normal, /Users/me/Desktop/fastq2/work/6e/48d4f61cae83ff903abceaadf715e5/SRR25452858_normal_1_val_1.fq.gz, /Users/me/Desktop/fastq2/work/6e/48d4f61cae83ff903abceaadf715e5/SRR25452858_normal_2_val_2.fq.gz]
  Analysis complete for SRR25452858_normal_1_val_1.fq.gz
  Analysis complete for SRR25452858_normal_2_val_2.fq.gz
  
  [SRR25452863_tumor, /Users/me/Desktop/fastq2/work/d3/0bd6515fcedfc060596b75dd90c28a/SRR25452863_tumor_1_val_1.fq.gz, /Users/me/Desktop/fastq2/work/d3/0bd6515fcedfc060596b75dd90c28a/SRR25452863_tumor_2_val_2.fq.gz]
  Analysis complete for SRR25452863_tumor_1_val_1.fq.gz
  Analysis complete for SRR25452863_tumor_2_val_2.fq.gz
  
  ```
  
  I am unsure if I am misunderstanding what Nextflow is capable of or how channels work. What I am trying to accomplish is trim adapters and QC result, align trimmed reads followed by indexing, and finally calculate mappings statistics. Any advice is appreciated.
  
  Here is my entire script:
  
  ```
  #!/usr/bin/env nextflow
  nextflow.enable.dsl = 2
  
  params.fastq = "$projectDir/fastq2/*_{1,2}*"
  params.index_dir = "$projectDir/RefGenome/BWAIndex"
  params.ref = "hg38.fa"
  params.out_dir = "$projectDir/Results"
  
  workflow {
  
      fastq_ch = Channel.fromFilePairs(params.fastq)
      index_ch = Channel.fromPath(params.index_dir)
      ref_ch = Channel.of(params.ref)
  
      trim_QC(fastq_ch)
      bwa_align(index_ch, ref_ch, trim_QC.out.trimmed_reads)
      mapping_stats(bwa_align.out.sorted_bam)
  }
  
  process trim_QC {
      debug true
      tag "$sample_id"
  
      publishDir = [
                      [
                          path: { "${params.out_dir}/trimmed/" },
                          mode: 'copy',
                          overwrite: false
                      ],
  
                      [
                          path: { "${params.out_dir}/stats/" },
                          mode: 'copy',
                          pattern: "*fastqc*",
                          overwrite: false
                      ]
                  ]
  
      input:
      tuple val(sample_id), path(fastq)
  
      output:
      tuple val("${sample_id}"), path("*_val_1.fq.gz"), path("*_val_2.fq.gz"), emit: trimmed_reads
      path("*fastqc*")
  
      script:
      def (read1, read2) = fastq
      """
      trim_galore --paired --fastqc ${read1} ${read2}
      """
  }
  
  process bwa_align {
      debug true
      tag "$sample_id"
  
      publishDir "${params.out_dir}/bwa_aligned/", mode: 'copy', overwrite: false
  
      input:
      path index_dir
      val ref
      tuple val(sample_id), path(read1), path(read2)
  
      output:
      path("${sample_id}_sorted.bam"), emit: sorted_bam
  path("${sample_id}_sorted.bam.bai"), emit: bam_index
  
      script:
      """
      bwa mem -t 2 ${index_dir}/${ref} ${read1} ${read2} | samtools sort -@2 -o "${sample_id}_sorted.bam" -
  samtools index ${sample_id}_sorted.bam > "${sample_id}_sorted.bam.bai"
      """
  }
  
  process mapping_stats {
      debug true
      tag "$sorted_bam"
  
      publishDir "${params.out_dir}/stats/", mode: 'copy', overwrite: false
  
      input:
      path(sorted_bam)
  
      output:
      path("${sorted_bam}_mapping_stats.txt")
  
      script:
      """
      samtools flagstat ${sorted_bam} > "${sorted_bam}_mapping_stats.txt"
      """
  }
  
  ```
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-13. Score: 2 | Accepted answer
  
  This is a result of nextflow channel types, [queue- and value-channels](https://www.nextflow.io/docs/latest/channel.html#channel-types). Nextflow will make fifo combinations of your channels to run `bwa_align` until either or both channel are empty. `index_ch` contains only one item, so once it is used there are no more valid combinations to start new `bwa_align` processes.
  
  Either [create](https://www.nextflow.io/docs/latest/channel.html#value) your `index_ch` as value channel, or keep the index out of a channel entirely and refer to it as a [parameter](https://www.nextflow.io/docs/latest/config.html#scope-params).
  
  > **Comments:**
  > **username_0**: Thanks for your response! I did as you suggested and used `Channel.value(params...)` for both my index and ref channels. While this did work I still seem to be missing something. While my `bwa_align` process is now aligning both sets of files, the `mapping_stats` still is 1 of 1. I was under the impression that any output from a process is a value channel, so this should work. Even more strange is this process only takes a single input. Do you have any advice for how to rectify this or what I am misunderstanding?  
  > **username_1**: Afaik a process will only emit a value channel if ALL its input channels were value channels, otherwise it will be a queue chanel. Make a new question if you have a new question, it is hard to debug your new issue without the exact code. The output you show seems to suggest you are using the `-resume` option, maybe this is messing with the correct execution after you changed the flow.  
  > **username_0**: Apologies, I was too impatient and responded prematurely. The executor updated to 2 of 2 for the `mapping_stats` following the completion of the second `bwa_align` process. Looks like everything is working as expected. Thanks again for your help!  
  
  ---
  
  