Answers.Body: <p>A better way would be to stage the files in a sub-directory, which would allow you to simply pass in a glob pattern to your tool. This can be achieved using the <code>path</code> input type and a string containing the <code>*</code> or <code>?</code> wildcards to control the names of staged files<sup><a href="https://www.nextflow.io/docs/stable/process.html#multiple-input-files" rel="noreferrer">1</a></sup>. Here, the files are named as the source files in a sub-directory called 'staged':</p>
  <pre><code>process PROCESS_B{
  
      input:
      path 'staged/*'
  
      script:
      &quot;&quot;&quot;
      mytool staged/*
      &quot;&quot;&quot;
  }
  </code></pre>
  <p>Note that you can also use the <code>stageAs</code> option for this, if a variable is somehow useful:</p>
  <pre><code>process PROCESS_B{
  
      input:
      path all_files_list, stageAs: 'staged/*'
  
      script:
      &quot;&quot;&quot;
      echo &quot;Staged ${all_files_list.size()} files...&quot;
      mytool staged/*
      &quot;&quot;&quot;
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-27T17:14:33.853Z
Answers.Text: Oh man, I can't believe I missed this in the docs. This is very elegant. Thank you!
Answers.UserDisplayName: DLJ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-27T14:17:41.497Z
Answers.LastActivityDate: 2024-09-27T14:17:41.497Z
Answers.UserDisplayName: Steve
Body: <p>Consider a hypothetical pipeline shown below, where many files need to be collected at one stage in the pipeline:</p>
  <p><img src="https://i.sstatic.net/Lhfs08Nd.png" alt="Hypothetical Pipeline" /></p>
  <p>It seems like the idiomatic way of accomplishing this is to pass <code>Process B</code> a list of filenames, which then passes those file names to the tool in question:</p>
  <pre><code>workflow {
      channel_with_many_files = channel.of(...)
  
      PROCESS_A(channel_with_many_files)
      
      all_files_together = PROCESS_A.out.collect()
  
      PROCESS_B(all_files_together)
  }
  </code></pre>
  <p>And then in the process B definition:</p>
  <pre><code>process PROCESS_B{
      input:
      path(all_files_list)
  
      output:
      path(single_output_file)
      
      script:
  
      &quot;&quot;&quot;
      ./mytool ${all_files_list}
      &quot;&quot;&quot;
  }
  </code></pre>
  <p>The problem with this, in my opinion, is that when <code>n</code> (number of files) grows to be very large (say &gt; 2000), the command that is generated in <code>PROCESS_B</code> is unweidly and a bit messy.</p>
  <p>Is this the way to deal with such a pattern in Nextflow, or is there a better way?</p>
  
Tags: <nextflow>
Title: Nextflow Best Practice When Collecting a Large Number of Files in One Step
ThreadText: # Nextflow Best Practice When Collecting a Large Number of Files in One Step
  
  (Asked by: username_0 on 2024-09-27)
  
  Consider a hypothetical pipeline shown below, where many files need to be collected at one stage in the pipeline:
  
  ![Hypothetical Pipeline](https://i.sstatic.net/Lhfs08Nd.png)
  
  It seems like the idiomatic way of accomplishing this is to pass `Process B` a list of filenames, which then passes those file names to the tool in question:
  
  ```
  workflow {
      channel_with_many_files = channel.of(...)
  
      PROCESS_A(channel_with_many_files)
      
      all_files_together = PROCESS_A.out.collect()
  
      PROCESS_B(all_files_together)
  }
  
  ```
  
  And then in the process B definition:
  
  ```
  process PROCESS_B{
      input:
      path(all_files_list)
  
      output:
      path(single_output_file)
      
      script:
  
      """
      ./mytool ${all_files_list}
      """
  }
  
  ```
  
  The problem with this, in my opinion, is that when `n` (number of files) grows to be very large (say > 2000), the command that is generated in `PROCESS_B` is unweidly and a bit messy.
  
  Is this the way to deal with such a pattern in Nextflow, or is there a better way?
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-27. Score: 5 | Accepted answer
  
  A better way would be to stage the files in a sub-directory, which would allow you to simply pass in a glob pattern to your tool. This can be achieved using the `path` input type and a string containing the `*` or `?` wildcards to control the names of staged files[1](https://www.nextflow.io/docs/stable/process.html#multiple-input-files). Here, the files are named as the source files in a sub-directory called 'staged':
  
  ```
  process PROCESS_B{
  
      input:
      path 'staged/*'
  
      script:
      """
      mytool staged/*
      """
  }
  
  ```
  
  Note that you can also use the `stageAs` option for this, if a variable is somehow useful:
  
  ```
  process PROCESS_B{
  
      input:
      path all_files_list, stageAs: 'staged/*'
  
      script:
      """
      echo "Staged ${all_files_list.size()} files..."
      mytool staged/*
      """
  }
  
  ```
  
  > **Comments:**
  > **username_0**: Oh man, I can't believe I missed this in the docs. This is very elegant. Thank you!  
  
  ---
  
  