Answers.Body: <p>My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with <a href="https://www.nextflow.io/docs/latest/process.html#multiple-input-files" rel="nofollow noreferrer">multiple input files</a> by using the <code>*</code> and <code>?</code> wildcards:</p>
  <blockquote>
  <p>When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.</p>
  <p>The target input file name can contain the * and ? wildcards, that can be used to control the name of staged files.</p>
  </blockquote>
  <p>Note that there is also the <code>stageAs</code> path option (see <a href="https://www.nextflow.io/docs/latest/process.html#input-of-type-path" rel="nofollow noreferrer">input of type 'path'</a>), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:</p>
  <pre><code>process samtools_merge {
  
      conda 'samtools=1.15.1'
      debug true
      stageInMode 'rellink'
  
      input:
      tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
  
      output:
      tuple val(sample), path(&quot;${sample}.bam&quot;)
  
      &quot;&quot;&quot;
      samtools merge &quot;<span class="math-container">${sample}.bam" $</span>{bam_files}
      ls -nl
      &quot;&quot;&quot;
  }
  
  workflow {
  
      Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
          | map { bam -&gt; tuple( bam.baseName, bam ) } \
          | groupTuple(by: 0, size: 2) \
          | samtools_merge
  }
  </code></pre>
  <p>Results:</p>
  <pre><code>N E X T F L O W  ~  version 22.04.0
  Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
  executor &gt;  local (3)
  [83/25654c] process &gt; samtools_merge (3) [100%] 3 of 3 ✔
  total 184
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample1.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample1.bam
  -rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam
  
  total 128
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample2.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample2.bam
  -rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam
  
  total 196
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample3.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample3.bam
  -rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam
  
  </code></pre>
  <p>If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special <a href="https://github.com/nextflow-io/nextflow/blob/v22.04.3/modules/nextflow/src/main/groovy/nextflow/extension/GroupKey.groovy" rel="nofollow noreferrer">groupKey</a> object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):</p>
  <pre><code>process samtools_sort {
  
      tag { bam.baseName }
  
      conda 'samtools=1.15.1'
  
      input:
      tuple val(sample), path(bam)
  
      output:
      tuple val(sample), path(&quot;${bam.baseName}.sorted.bam&quot;)
  
      &quot;&quot;&quot;
      samtools sort -o &quot;<span class="math-container">${bam.baseName}.sorted.bam" "$</span>{bam}&quot;
      &quot;&quot;&quot;
  }
  
  process samtools_merge {
  
      tag { sample }
  
      conda 'samtools=1.15.1'
  
      input:
      tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
  
      output:
      tuple val(sample), path(&quot;${sample}.bam&quot;)
  
      &quot;&quot;&quot;
      samtools merge &quot;<span class="math-container">${sample}.bam" $</span>{bam_files}
      &quot;&quot;&quot;
  }
  </code></pre>
  <pre><code>workflow {
  
      Channel.fromPath( params.bam_files ) \
          | map { bam -&gt; tuple( bam.baseName, bam ) } \
          | groupTuple() \
          | map { sample, files -&gt; tuple( groupKey(sample, files.size()), files ) } \
          | transpose() \
          | set { bam_files }
  
      samtools_sort( bam_files ) \
          | groupTuple() \
          | samtools_merge \
          | view()
  }
  </code></pre>
  <p>Results:</p>
  <pre><code>$ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
  N E X T F L O W  ~  version 22.04.0
  Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
  [04/643129] Submitted process &gt; samtools_sort (sample3)
  [1a/93a939] Submitted process &gt; samtools_sort (sample1)
  [93/fde0a0] Submitted process &gt; samtools_sort (sample3)
  [db/208bac] Submitted process &gt; samtools_sort (sample3)
  [8f/f2a231] Submitted process &gt; samtools_sort (sample2)
  [f3/8f0f9f] Submitted process &gt; samtools_sort (sample2)
  [bb/cca23e] Submitted process &gt; samtools_merge (sample1)
  [sample1, /home/steve/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
  [ff/bd3a2a] Submitted process &gt; samtools_merge (sample2)
  [sample2, /home/steve/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
  [94/a4f8eb] Submitted process &gt; samtools_merge (sample3)
  [sample3, /home/steve/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]
  </code></pre>
  <p>Note that the <a href="https://www.nextflow.io/docs/latest/process.html#input-of-generic-values" rel="nofollow noreferrer">val</a> specified as input to 'samtools_sort' is an object of class <code>nextflow.extension.GroupKey</code>. In the example above, the input to 'samtools_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:</p>
  <pre><code>    ...
  
      samtools_sort( bam_files ) \
          | groupTuple() \
          | map { sample_key, files -&gt; tuple( sample_key.toString(), files ) } \
          | samtools_merge \
          | view()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-25T14:12:51.580Z
Answers.Text: Thank you so much for this very comprehensive answer! I had totally missed that section of the `input` documentation. And as ATpoint notes, this is the second time you help me on a Nextflow question, this has been extremely helpful!
Answers.UserDisplayName: Alexlok
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-25T06:16:47.527Z
Answers.LastActivityDate: 2022-05-25T06:16:47.527Z
Answers.UserDisplayName: Steve
Body: <p>I have a process that emits BAM files with a key containing the sample name:</p>
  <pre><code>process source{
    output:
      tuple val(sample), path(&quot;sample.bam&quot;) into ch
  
    [...]
  
  }
  </code></pre>
  <p>1st question: I have several files per <code>sample</code>, let's assume 2 for now. I want to merge them together with <a href="http://www.htslib.org/doc/samtools-merge.html" rel="nofollow noreferrer"><code>samtools merge</code></a>. I can regroup that channel with <a href="https://www.nextflow.io/docs/latest/operator.html#grouptuple" rel="nofollow noreferrer"><code>groupTuple()</code></a>, but how can I recover them in the next process? Considering they all have the same name:</p>
  <pre><code>grouped_ch = ch.groupTuple(by: 0, size: 2)
  
  process merge{
    input:
      grouped_ch
  
    shell:
    '''
    samtool merge sample.bam sample.bam &gt; !{sample}.bam
    '''
  }
  </code></pre>
  <p>Is there perhaps a way in the <code>input:</code> block to say that I expect a list, and name the list components?</p>
  <p>2nd question: In practice, I have a variable number of &quot;sample.bam&quot; files for each <code>sample</code> value, does that change the way to define the <code>input:</code>? It also appears I will need to use the built-in <code>groupKey()</code>, is there documentation available? I have read <a href="https://github.com/nextflow-io/nextflow/issues/796#issuecomment-407108667" rel="nofollow noreferrer">this Github issue</a>, but it would help if a general description was available.</p>
  
Tags: <nextflow>
Title: recover files after groupTuple
ThreadText: # recover files after groupTuple
  
  (Asked by: username_0 on 2022-05-24)
  
  I have a process that emits BAM files with a key containing the sample name:
  
  ```
  process source{
    output:
      tuple val(sample), path("sample.bam") into ch
  
    [...]
  
  }
  
  ```
  
  1st question: I have several files per `sample`, let's assume 2 for now. I want to merge them together with [`samtools merge`](http://www.htslib.org/doc/samtools-merge.html). I can regroup that channel with [`groupTuple()`](https://www.nextflow.io/docs/latest/operator.html#grouptuple), but how can I recover them in the next process? Considering they all have the same name:
  
  ```
  grouped_ch = ch.groupTuple(by: 0, size: 2)
  
  process merge{
    input:
      grouped_ch
  
    shell:
    '''
    samtool merge sample.bam sample.bam > !{sample}.bam
    '''
  }
  
  ```
  
  Is there perhaps a way in the `input:` block to say that I expect a list, and name the list components?
  
  2nd question: In practice, I have a variable number of "sample.bam" files for each `sample` value, does that change the way to define the `input:`? It also appears I will need to use the built-in `groupKey()`, is there documentation available? I have read [this Github issue](https://github.com/nextflow-io/nextflow/issues/796#issuecomment-407108667), but it would help if a general description was available.
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-25. Score: 3 | Accepted answer
  
  My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with [multiple input files](https://www.nextflow.io/docs/latest/process.html#multiple-input-files) by using the `*` and `?` wildcards:
  
  > When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.
  >
  > The target input file name can contain the \* and ? wildcards, that can be used to control the name of staged files.
  
  Note that there is also the `stageAs` path option (see [input of type 'path'](https://www.nextflow.io/docs/latest/process.html#input-of-type-path)), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:
  
  ```
  process samtools_merge {
  
      conda 'samtools=1.15.1'
      debug true
      stageInMode 'rellink'
  
      input:
      tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
  
      output:
      tuple val(sample), path("${sample}.bam")
  
      """
      samtools merge "${sample}.bam" ${bam_files}
      ls -nl
      """
  }
  
  workflow {
  
      Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
          | map { bam -> tuple( bam.baseName, bam ) } \
          | groupTuple(by: 0, size: 2) \
          | samtools_merge
  }
  
  ```
  
  Results:
  
  ```
  N E X T F L O W  ~  version 22.04.0
  Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
  executor >  local (3)
  [83/25654c] process > samtools_merge (3) [100%] 3 of 3 ✔
  total 184
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample1.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample1.bam
  -rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam
  
  total 128
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample2.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample2.bam
  -rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam
  
  total 196
  lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample3.bam
  lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample3.bam
  -rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam
  
  ```
  
  If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special [groupKey](https://github.com/nextflow-io/nextflow/blob/v22.04.3/modules/nextflow/src/main/groovy/nextflow/extension/GroupKey.groovy) object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools\_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools\_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):
  
  ```
  process samtools_sort {
  
      tag { bam.baseName }
  
      conda 'samtools=1.15.1'
  
      input:
      tuple val(sample), path(bam)
  
      output:
      tuple val(sample), path("${bam.baseName}.sorted.bam")
  
      """
      samtools sort -o "${bam.baseName}.sorted.bam" "${bam}"
      """
  }
  
  process samtools_merge {
  
      tag { sample }
  
      conda 'samtools=1.15.1'
  
      input:
      tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')
  
      output:
      tuple val(sample), path("${sample}.bam")
  
      """
      samtools merge "${sample}.bam" ${bam_files}
      """
  }
  
  ```
  
  ```
  workflow {
  
      Channel.fromPath( params.bam_files ) \
          | map { bam -> tuple( bam.baseName, bam ) } \
          | groupTuple() \
          | map { sample, files -> tuple( groupKey(sample, files.size()), files ) } \
          | transpose() \
          | set { bam_files }
  
      samtools_sort( bam_files ) \
          | groupTuple() \
          | samtools_merge \
          | view()
  }
  
  ```
  
  Results:
  
  ```
  $ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
  N E X T F L O W  ~  version 22.04.0
  Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
  [04/643129] Submitted process > samtools_sort (sample3)
  [1a/93a939] Submitted process > samtools_sort (sample1)
  [93/fde0a0] Submitted process > samtools_sort (sample3)
  [db/208bac] Submitted process > samtools_sort (sample3)
  [8f/f2a231] Submitted process > samtools_sort (sample2)
  [f3/8f0f9f] Submitted process > samtools_sort (sample2)
  [bb/cca23e] Submitted process > samtools_merge (sample1)
  [sample1, /home/username_1/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
  [ff/bd3a2a] Submitted process > samtools_merge (sample2)
  [sample2, /home/username_1/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
  [94/a4f8eb] Submitted process > samtools_merge (sample3)
  [sample3, /home/username_1/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]
  
  ```
  
  Note that the [val](https://www.nextflow.io/docs/latest/process.html#input-of-generic-values) specified as input to 'samtools\_sort' is an object of class `nextflow.extension.GroupKey`. In the example above, the input to 'samtools\_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:
  
  ```
      ...
  
      samtools_sort( bam_files ) \
          | groupTuple() \
          | map { sample_key, files -> tuple( sample_key.toString(), files ) } \
          | samtools_merge \
          | view()
  
  ```
  
  > **Comments:**
  > **username_0**: Thank you so much for this very comprehensive answer! I had totally missed that section of the `input` documentation. And as ATpoint notes, this is the second time you help me on a Nextflow question, this has been extremely helpful!  
  
  ---
  
  