Answers.Body: <p>I think the answer to your question is the <a href="https://www.nextflow.io/docs/latest/process.html#input-repeaters-each" rel="nofollow noreferrer"><code>each</code> input repeaters</a>. You can also check <a href="https://www.nextflow.io/docs/latest/faq.html#how-do-i-iterate-over-a-process-n-times" rel="nofollow noreferrer">Nextflow FAQ</a> for the question on how to iterate over a process or a file.</p>
  <p>Not in front of a PC now but you could try something like <code>each lineage_dataset</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T09:32:53.053Z
Answers.LastActivityDate: 2023-03-07T09:32:53.053Z
Answers.UserDisplayName: user324810
Body: <p><em>This question was also asked on <a href="https://stackoverflow.com/questions/75076433/run-nextflow-pipeline-multiple-time-depending-on-length-of-a-list">Stack Overflow</a></em></p>
  <p>I have a Nextflow pipeline that runs a BUSCO search on input data, returns the output and then creates a report based on the output. One of my parameters is the lineage dataset. I would like to have my pipeline run multiple times (preferably at the same time) based on how long my list of lineage datasets is. For example, this is my nextflow.config file. The goal is that my pipeline would run twice in this case. If there are n lineages specified I would like it to run n times.</p>
  <pre><code>params { 
      inputFilePath           = &quot;$PWD/input_data/test_data.fasta&quot;
      lineage_datasets        = [
          &quot;fungi_odb10&quot;,
          &quot;microsporidia_odb10&quot;
          ]
      augustus_species        = &quot;fumigatus&quot;
      mode                    = &quot;geno&quot;
      
      threads                 = 16
      help                    = false
      outdir {
          main                = &quot;<span class="math-container">$PWD/results"
      intermediate        = "$</span>PWD/busco_outputs&quot;
      }
  }
  </code></pre>
  <p>The first part of my busco process looks like this:</p>
  <pre><code>  publishDir params.outdir.intermediate, mode: 'copy'
  
    input:
      path inputFile
      string lineage_dataset
    
    output:
      path 'hap1/*hap1.txt'
  </code></pre>
  <p>The idea is that each of the summary.txt files will be collected inside of my outdir.intermediate folder which is where my createReport process will target. For context, this is my createReport process:</p>
  <pre><code>process createReport {
  
    publishDir params.outdir.main, mode: 'copy'
  
    input: 
      path 'busco_outputs/*'
  
    output:
      path 'report.html'
  
    script:
      &quot;&quot;&quot;
      source &quot;$launchDir/venv/bin/activate&quot;
      report.py &gt; report.html
      &quot;&quot;&quot; 
  }
  </code></pre>
  <p>I have tried the following in my workflow process but it does not seem to work:</p>
  <pre><code>workflow {
    runBuscoChannel = Channel.fromPath(params.inputFilePath).cross(Channel.of(params.lineage_datasets)) 
    | mix 
    | collect 
    | set { outputs }
  
    createReport(outputs)
  }
  </code></pre>
  <p>When I try to run the pipeline it does not run properly</p>
  <p>Does anyone have any ideas of how I could change my file to allow for this? Thanks very much!</p>
  
Tags: <bash><nextflow>
Title: Run nextflow pipeline multiple times depending on length of a list
ThreadText: # Run nextflow pipeline multiple times depending on length of a list
  
  (Asked by: username_0 on 2023-01-10)
  
  *This question was also asked on [Stack Overflow](https://stackoverflow.com/questions/75076433/run-nextflow-pipeline-multiple-time-depending-on-length-of-a-list)*
  
  I have a Nextflow pipeline that runs a BUSCO search on input data, returns the output and then creates a report based on the output. One of my parameters is the lineage dataset. I would like to have my pipeline run multiple times (preferably at the same time) based on how long my list of lineage datasets is. For example, this is my nextflow.config file. The goal is that my pipeline would run twice in this case. If there are n lineages specified I would like it to run n times.
  
  ```
  params { 
      inputFilePath           = "$PWD/input_data/test_data.fasta"
      lineage_datasets        = [
          "fungi_odb10",
          "microsporidia_odb10"
          ]
      augustus_species        = "fumigatus"
      mode                    = "geno"
      
      threads                 = 16
      help                    = false
      outdir {
          main                = "$PWD/results"
      intermediate        = "$PWD/busco_outputs"
      }
  }
  
  ```
  
  The first part of my busco process looks like this:
  
  ```
    publishDir params.outdir.intermediate, mode: 'copy'
  
    input:
      path inputFile
      string lineage_dataset
    
    output:
      path 'hap1/*hap1.txt'
  
  ```
  
  The idea is that each of the summary.txt files will be collected inside of my outdir.intermediate folder which is where my createReport process will target. For context, this is my createReport process:
  
  ```
  process createReport {
  
    publishDir params.outdir.main, mode: 'copy'
  
    input: 
      path 'busco_outputs/*'
  
    output:
      path 'report.html'
  
    script:
      """
      source "$launchDir/venv/bin/activate"
      report.py > report.html
      """ 
  }
  
  ```
  
  I have tried the following in my workflow process but it does not seem to work:
  
  ```
  workflow {
    runBuscoChannel = Channel.fromPath(params.inputFilePath).cross(Channel.of(params.lineage_datasets)) 
    | mix 
    | collect 
    | set { outputs }
  
    createReport(outputs)
  }
  
  ```
  
  When I try to run the pipeline it does not run properly
  
  Does anyone have any ideas of how I could change my file to allow for this? Thanks very much!
  
  **Tags:** bash, nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-07. Score: 2
  
  I think the answer to your question is the [`each` input repeaters](https://www.nextflow.io/docs/latest/process.html#input-repeaters-each). You can also check [Nextflow FAQ](https://www.nextflow.io/docs/latest/faq.html#how-do-i-iterate-over-a-process-n-times) for the question on how to iterate over a process or a file.
  
  Not in front of a PC now but you could try something like `each lineage_dataset`.
  
  
  ---
  
  