Answers.Body: <p>You don't need to create processes in a loop when you have channels. The <a href="https://www.nextflow.io/docs/latest/process.html#input-repeaters" rel="nofollow noreferrer">each qualifier</a> allows you to repeat the execution of a process for each item in a collection. For example, the contents of 'test.nf':</p>
  <pre><code>some_inputs = Channel.of( 'prot', 'dna', 'rna' )
  some_parameters = Channel.of( 5, 3, 2, 7, 1 )
  
  process foo {
  
      tag { &quot;task: <span class="math-container">${task.index} - $</span>{x},${y}&quot; }
  
      input:
      val x from some_inputs
      each y from some_parameters
  
      output:
      tuple val(x), val(y) into receivers
  
      &quot;&quot;&quot;
      echo &quot;x: <span class="math-container">${x}; y: $</span>{y}&quot;
      &quot;&quot;&quot;
  }
  
  results = receivers.branch { x, y -&gt;
      
      five: y == 5
          return tuple( x, y )
  
      three: y == 3
          return tuple( x, y )
  
      two: y == 2
          return tuple( x, y )
  
      seven: y == 7
          return tuple( x, y )
      
      one: y == 1
          return tuple( x, y )
  }
  
  results.five.view { &quot;My result for param 5 is: <span class="math-container">$it" }
  results.three.view { "My result for param 3 is: $it" }
  results.two.view { "My result for param 2 is: $it" }
  results.seven.view { "My result for param 7 is: $it" }
  results.one.view { "My result for param 1 is: $</span>it&quot; }
  </code></pre>
  <p>Then running like:</p>
  <pre><code>nextflow run -ansi-log false test.nf
  </code></pre>
  <p>Results:</p>
  <pre><code>N E X T F L O W  ~  version 20.10.0
  Launching `test.nf` [fabulous_franklin] - revision: 14c3290d2b
  [6e/c12ce7] Submitted process &gt; foo (task: 7 - dna,3)
  [93/616f93] Submitted process &gt; foo (task: 4 - prot,7)
  [27/0bc3c4] Submitted process &gt; foo (task: 3 - prot,2)
  [0d/0d0b51] Submitted process &gt; foo (task: 9 - dna,7)
  [d4/f56d18] Submitted process &gt; foo (task: 6 - dna,5)
  [b0/d1535f] Submitted process &gt; foo (task: 8 - dna,2)
  [13/0a660a] Submitted process &gt; foo (task: 5 - prot,1)
  [f0/1c2f57] Submitted process &gt; foo (task: 2 - prot,3)
  [3f/3e893d] Submitted process &gt; foo (task: 1 - prot,5)
  [19/2921de] Submitted process &gt; foo (task: 10 - dna,1)
  My result for param 3 is: [dna, 3]
  My result for param 7 is: [prot, 7]
  My result for param 7 is: [dna, 7]
  [a7/953a2b] Submitted process &gt; foo (task: 11 - rna,3)
  [1c/0fd4ef] Submitted process &gt; foo (task: 12 - rna,5)
  My result for param 1 is: [prot, 1]
  [39/d21adf] Submitted process &gt; foo (task: 14 - rna,7)
  My result for param 5 is: [dna, 5]
  My result for param 2 is: [prot, 2]
  My result for param 2 is: [dna, 2]
  [f6/8509a7] Submitted process &gt; foo (task: 15 - rna,1)
  My result for param 5 is: [prot, 5]
  [6f/cee0ae] Submitted process &gt; foo (task: 13 - rna,2)
  My result for param 3 is: [rna, 3]
  My result for param 1 is: [dna, 1]
  My result for param 2 is: [rna, 2]
  My result for param 3 is: [prot, 3]
  My result for param 1 is: [rna, 1]
  My result for param 5 is: [rna, 5]
  My result for param 7 is: [rna, 7]
  </code></pre>
  <p>The answer to your bonus question is to use the <code>task.index</code> variable in your script. I used this to <a href="https://www.nextflow.io/docs/latest/process.html#tag" rel="nofollow noreferrer">tag</a> your job, but it can be used anywhere in the process declaration. You likely won't need this variable though.</p>
  <p>Usually, using the <a href="https://www.nextflow.io/docs/latest/operator.html#branch" rel="nofollow noreferrer">branch operator</a> is enough for most use cases. However,it's possible to implement your own branching function. The following uses the new DSL, DSL2, to automatically fork the 'receivers' channel:</p>
  <pre><code>nextflow.enable.dsl=2
  
  process foo {
  
      tag { &quot;task: <span class="math-container">${task.index} - $</span>{x},${y}&quot; }
  
      input:
      val x
      each y
  
      output:
      tuple val(x), val(y)
  
      &quot;&quot;&quot;
      echo &quot;x: <span class="math-container">${x}; y: $</span>{y}&quot;
      &quot;&quot;&quot;
  }
  
  def branchByElement( input_ch, elements, by=0 ) {
  
      def targets = [:]
      for( element in elements ) {
          def key = element
          targets[ key ] = { input_ch.filter { it[by] == key } }
      }
      return targets
  }
  
  workflow {
  
      some_inputs = [ 'prot', 'dna', 'rna' ]
      some_parameters = [ 5, 3, 2, 7, 1 ]
  
      inputs_ch = Channel.fromList( some_inputs )
      parameters_ch = Channel.fromList( some_parameters )
  
      foo( inputs_ch, parameters_ch )
  
      parameter_results = branchByElement( foo.out, some_parameters, by=1 )
      input_results = branchByElement( foo.out, some_inputs, by=0 )
  
      parameter_results[7].call().view { &quot;Param 7: <span class="math-container">$it" }
      input_results.dna().view { "Input dna: $</span>it&quot; }
  }
  </code></pre>
  <p>Results:</p>
  <pre><code>N E X T F L O W  ~  version 20.10.0
  Launching `test.nf` [distraught_meninsky] - revision: 04df251752
  [bb/32325f] Submitted process &gt; foo (task: 4 - prot,7)
  [a0/19fdf5] Submitted process &gt; foo (task: 8 - dna,2)
  [e9/4c1d09] Submitted process &gt; foo (task: 6 - dna,5)
  [a3/dcb002] Submitted process &gt; foo (task: 2 - prot,3)
  [db/7e612e] Submitted process &gt; foo (task: 5 - prot,1)
  [cd/dfa84e] Submitted process &gt; foo (task: 9 - dna,7)
  [d4/6a85f2] Submitted process &gt; foo (task: 7 - dna,3)
  [f0/8e8bf4] Submitted process &gt; foo (task: 1 - prot,5)
  [89/f20a28] Submitted process &gt; foo (task: 3 - prot,2)
  Input dna: [dna, 2]
  Param 7: [prot, 7]
  Input dna: [dna, 5]
  [53/056fe4] Submitted process &gt; foo (task: 10 - dna,1)
  [f3/a61abf] Submitted process &gt; foo (task: 12 - rna,3)
  Input dna: [dna, 7]
  [f9/a568a8] Submitted process &gt; foo (task: 13 - rna,2)
  Param 7: [dna, 7]
  [06/72bb6c] Submitted process &gt; foo (task: 11 - rna,5)
  Input dna: [dna, 3]
  [a3/6d80d8] Submitted process &gt; foo (task: 14 - rna,7)
  [ac/acdb5a] Submitted process &gt; foo (task: 15 - rna,1)
  Param 7: [rna, 7]
  Input dna: [dna, 1]
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T14:51:15.133Z
Answers.Text: I think `each` is even more elegant than combine, do you also have a more elegant way to split the resulting channel than `branch` ?
Answers.UserDisplayName: Pallie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T14:02:01.260Z
Answers.Text: @Pallie I think there is using a function and the new DSL2 - please see my update above.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T14:28:38.817Z
Answers.LastActivityDate: 2021-04-14T14:00:36.763Z
Answers.LastEditDate: 2021-04-14T14:00:36.763Z
Answers.UserDisplayName: Steve
Answers.Body: <p>Use channels, this is what nextflow's entire design is based on. I'm honestly surprised that spawning 5 processes with the same name in a loop even works.</p>
  <p>Turn your parameters into regular channels, use the <code>combine</code> operator (<a href="https://www.nextflow.io/docs/latest/operator.html#combine" rel="nofollow noreferrer">https://www.nextflow.io/docs/latest/operator.html#combine</a>) to transform the 2 parameter channels into a channel that has each combination of params/inputs. Feed this channel into the  <code>foo</code>  process. Afterward use <code>branch</code> (<a href="https://www.nextflow.io/docs/latest/operator.html#branch" rel="nofollow noreferrer">https://www.nextflow.io/docs/latest/operator.html#branch</a>) to split your <code>foo</code> output channel into 5 different channels.</p>
  <pre><code>typechan = Channel.from(5,3,2,7,1)
  numchan = Channel.from('prot','dna', 'rna')
  
  cartesian = numchan.combine(typechan)
  
  process foo {
    input:
    val numtype from cartesian
    output:
    val numtype into bar 
    script:
    &quot;&quot;&quot;
    echo $numtype
    &quot;&quot;&quot;
  }
  
  bar.branch {
    one: it[1] == 1
    two: it[1] == 2
    three: it[1] == 3 
    five: it[1] == 5
    seven : it[1] == 7
  }.set {result}
  result.one.view()
  <span class="math-container">```</span>
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T11:25:08.340Z
Answers.LastActivityDate: 2021-04-13T11:25:08.340Z
Answers.UserDisplayName: Pallie
Body: <p>I have set of ~5 processes which perform very similar tasks such that their differences can be parameterized. Rather than putting 5 items through some channel (which seems complicated), or defining 5 separate processes, it appears to be possible in Nextflow to declare the processes in a for loop, and output into an array of channels.</p>
  <p>Below is a prototypical example and it seems to work as intended. (I understand this could be done differently with some channel cross product operator but I am just giving this as an example of a process in a for loop.)</p>
  <p>Is there anything wrong with doing this? Is there a better way to do it?</p>
  <p>Bonus question - the name of the process below is &quot;foo&quot;. I would like to have the index reflected in the name i.e. &quot;foo_$i&quot; but this doesn't seem possible by default - any way to do this??</p>
  <pre><code>some_inputs = ['prot','dna', 'rna']
  
  some_parameters = [5,3,2,7,1]
  
  // Set up an array of channels to output to
  receivers = new ArrayList&lt;Channel&gt;()
  for (i = 0; i &lt; 5; i++) {
      receivers.add(Channel.create())
  }
  
  // Iterate 5 times to create 5 processes each with different parameters
  for (i = 0; i &lt; 5; i++) {
  
      // WOW - process can be defined inside a for loop!!
      process &quot;foo&quot; {
  
        // Extract parameters here so correct $i value is used
        p = some_parameters[i]
  
        input:
        val x from Channel.from(some_inputs)
  
        output:
        val &quot;<span class="math-container">$x $</span>p&quot; into receivers[i] // Note that correct index $i seems to be used here
  
        &quot;&quot;&quot;
        &quot;&quot;&quot;
      }
  }
  
  // Confirm that each of the processes outputted independently with the correct parameter
  receivers[0].view { &quot;Received[0]: <span class="math-container">$it" }
  receivers[1].view { "Received[1]: $it" }
  receivers[2].view { "Received[2]: $it" }
  receivers[3].view { "Received[3]: $it" }
  receivers[4].view { "Received[4]: $</span>it&quot; }
  </code></pre>
  <p>Output - note that the parameter is consistent within each input channel i.e. working correctly:</p>
  <pre><code>N E X T F L O W  ~  version 20.10.0
  Launching `play.nf` [fabulous_keller] - revision: c89e2b2281
  executor &gt;  local (15)
  [10/b477ec] process &gt; foo (3) [100%] 3 of 3 ✔
  [16/8282ba] process &gt; foo (2) [100%] 3 of 3 ✔
  [db/f7327e] process &gt; foo (1) [100%] 3 of 3 ✔
  [fb/886f6f] process &gt; foo (1) [100%] 3 of 3 ✔
  [46/c34ed3] process &gt; foo (1) [100%] 3 of 3 ✔
  Received[1]: rna 3
  Received[1]: prot 3
  Received[4]: rna 1
  Received[2]: rna 2
  Received[0]: dna 5
  Received[2]: dna 2
  Received[2]: prot 2
  Received[0]: prot 5
  Received[3]: rna 7
  Received[0]: rna 5
  Received[1]: dna 3
  Received[3]: dna 7
  Received[4]: dna 1
  Received[3]: prot 7
  Received[4]: prot 1
  </code></pre>
  
Tags: <nextflow>
Title: Nextflow: Can I parameterize similar processes inside a for loop?
ThreadText: # Nextflow: Can I parameterize similar processes inside a for loop?
  
  (Asked by: username_0 on 2021-04-13)
  
  I have set of ~5 processes which perform very similar tasks such that their differences can be parameterized. Rather than putting 5 items through some channel (which seems complicated), or defining 5 separate processes, it appears to be possible in Nextflow to declare the processes in a for loop, and output into an array of channels.
  
  Below is a prototypical example and it seems to work as intended. (I understand this could be done differently with some channel cross product operator but I am just giving this as an example of a process in a for loop.)
  
  Is there anything wrong with doing this? Is there a better way to do it?
  
  Bonus question - the name of the process below is "foo". I would like to have the index reflected in the name i.e. "foo\_$i" but this doesn't seem possible by default - any way to do this??
  
  ```
  some_inputs = ['prot','dna', 'rna']
  
  some_parameters = [5,3,2,7,1]
  
  // Set up an array of channels to output to
  receivers = new ArrayList<Channel>()
  for (i = 0; i < 5; i++) {
      receivers.add(Channel.create())
  }
  
  // Iterate 5 times to create 5 processes each with different parameters
  for (i = 0; i < 5; i++) {
  
      // WOW - process can be defined inside a for loop!!
      process "foo" {
  
        // Extract parameters here so correct $i value is used
        p = some_parameters[i]
  
        input:
        val x from Channel.from(some_inputs)
  
        output:
        val "$x $p" into receivers[i] // Note that correct index $i seems to be used here
  
        """
        """
      }
  }
  
  // Confirm that each of the processes outputted independently with the correct parameter
  receivers[0].view { "Received[0]: $it" }
  receivers[1].view { "Received[1]: $it" }
  receivers[2].view { "Received[2]: $it" }
  receivers[3].view { "Received[3]: $it" }
  receivers[4].view { "Received[4]: $it" }
  
  ```
  
  Output - note that the parameter is consistent within each input channel i.e. working correctly:
  
  ```
  N E X T F L O W  ~  version 20.10.0
  Launching `play.nf` [fabulous_keller] - revision: c89e2b2281
  executor >  local (15)
  [10/b477ec] process > foo (3) [100%] 3 of 3 ✔
  [16/8282ba] process > foo (2) [100%] 3 of 3 ✔
  [db/f7327e] process > foo (1) [100%] 3 of 3 ✔
  [fb/886f6f] process > foo (1) [100%] 3 of 3 ✔
  [46/c34ed3] process > foo (1) [100%] 3 of 3 ✔
  Received[1]: rna 3
  Received[1]: prot 3
  Received[4]: rna 1
  Received[2]: rna 2
  Received[0]: dna 5
  Received[2]: dna 2
  Received[2]: prot 2
  Received[0]: prot 5
  Received[3]: rna 7
  Received[0]: rna 5
  Received[1]: dna 3
  Received[3]: dna 7
  Received[4]: dna 1
  Received[3]: prot 7
  Received[4]: prot 1
  
  ```
  
  **Tags:** nextflow
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-04-13. Score: 4 | Accepted answer
  
  Use channels, this is what nextflow's entire design is based on. I'm honestly surprised that spawning 5 processes with the same name in a loop even works.
  
  Turn your parameters into regular channels, use the `combine` operator (<https://www.nextflow.io/docs/latest/operator.html#combine>) to transform the 2 parameter channels into a channel that has each combination of params/inputs. Feed this channel into the `foo` process. Afterward use `branch` (<https://www.nextflow.io/docs/latest/operator.html#branch>) to split your `foo` output channel into 5 different channels.
  
  ```
  typechan = Channel.from(5,3,2,7,1)
  numchan = Channel.from('prot','dna', 'rna')
  
  cartesian = numchan.combine(typechan)
  
  process foo {
    input:
    val numtype from cartesian
    output:
    val numtype into bar 
    script:
    """
    echo $numtype
    """
  }
  
  bar.branch {
    one: it[1] == 1
    two: it[1] == 2
    three: it[1] == 3 
    five: it[1] == 5
    seven : it[1] == 7
  }.set {result}
  result.one.view()
  ```
  
  ```
  
  
  ---
  
  ### Answer by username_1 on 2021-04-13. Score: 3
  
  You don't need to create processes in a loop when you have channels. The [each qualifier](https://www.nextflow.io/docs/latest/process.html#input-repeaters) allows you to repeat the execution of a process for each item in a collection. For example, the contents of 'test.nf':
  
  ```
  some_inputs = Channel.of( 'prot', 'dna', 'rna' )
  some_parameters = Channel.of( 5, 3, 2, 7, 1 )
  
  process foo {
  
      tag { "task: ${task.index} - ${x},${y}" }
  
      input:
      val x from some_inputs
      each y from some_parameters
  
      output:
      tuple val(x), val(y) into receivers
  
      """
      echo "x: ${x}; y: ${y}"
      """
  }
  
  results = receivers.branch { x, y ->
      
      five: y == 5
          return tuple( x, y )
  
      three: y == 3
          return tuple( x, y )
  
      two: y == 2
          return tuple( x, y )
  
      seven: y == 7
          return tuple( x, y )
      
      one: y == 1
          return tuple( x, y )
  }
  
  results.five.view { "My result for param 5 is: $it" }
  results.three.view { "My result for param 3 is: $it" }
  results.two.view { "My result for param 2 is: $it" }
  results.seven.view { "My result for param 7 is: $it" }
  results.one.view { "My result for param 1 is: $it" }
  
  ```
  
  Then running like:
  
  ```
  nextflow run -ansi-log false test.nf
  
  ```
  
  Results:
  
  ```
  N E X T F L O W  ~  version 20.10.0
  Launching `test.nf` [fabulous_franklin] - revision: 14c3290d2b
  [6e/c12ce7] Submitted process > foo (task: 7 - dna,3)
  [93/616f93] Submitted process > foo (task: 4 - prot,7)
  [27/0bc3c4] Submitted process > foo (task: 3 - prot,2)
  [0d/0d0b51] Submitted process > foo (task: 9 - dna,7)
  [d4/f56d18] Submitted process > foo (task: 6 - dna,5)
  [b0/d1535f] Submitted process > foo (task: 8 - dna,2)
  [13/0a660a] Submitted process > foo (task: 5 - prot,1)
  [f0/1c2f57] Submitted process > foo (task: 2 - prot,3)
  [3f/3e893d] Submitted process > foo (task: 1 - prot,5)
  [19/2921de] Submitted process > foo (task: 10 - dna,1)
  My result for param 3 is: [dna, 3]
  My result for param 7 is: [prot, 7]
  My result for param 7 is: [dna, 7]
  [a7/953a2b] Submitted process > foo (task: 11 - rna,3)
  [1c/0fd4ef] Submitted process > foo (task: 12 - rna,5)
  My result for param 1 is: [prot, 1]
  [39/d21adf] Submitted process > foo (task: 14 - rna,7)
  My result for param 5 is: [dna, 5]
  My result for param 2 is: [prot, 2]
  My result for param 2 is: [dna, 2]
  [f6/8509a7] Submitted process > foo (task: 15 - rna,1)
  My result for param 5 is: [prot, 5]
  [6f/cee0ae] Submitted process > foo (task: 13 - rna,2)
  My result for param 3 is: [rna, 3]
  My result for param 1 is: [dna, 1]
  My result for param 2 is: [rna, 2]
  My result for param 3 is: [prot, 3]
  My result for param 1 is: [rna, 1]
  My result for param 5 is: [rna, 5]
  My result for param 7 is: [rna, 7]
  
  ```
  
  The answer to your bonus question is to use the `task.index` variable in your script. I used this to [tag](https://www.nextflow.io/docs/latest/process.html#tag) your job, but it can be used anywhere in the process declaration. You likely won't need this variable though.
  
  Usually, using the [branch operator](https://www.nextflow.io/docs/latest/operator.html#branch) is enough for most use cases. However,it's possible to implement your own branching function. The following uses the new DSL, DSL2, to automatically fork the 'receivers' channel:
  
  ```
  nextflow.enable.dsl=2
  
  process foo {
  
      tag { "task: ${task.index} - ${x},${y}" }
  
      input:
      val x
      each y
  
      output:
      tuple val(x), val(y)
  
      """
      echo "x: ${x}; y: ${y}"
      """
  }
  
  def branchByElement( input_ch, elements, by=0 ) {
  
      def targets = [:]
      for( element in elements ) {
          def key = element
          targets[ key ] = { input_ch.filter { it[by] == key } }
      }
      return targets
  }
  
  workflow {
  
      some_inputs = [ 'prot', 'dna', 'rna' ]
      some_parameters = [ 5, 3, 2, 7, 1 ]
  
      inputs_ch = Channel.fromList( some_inputs )
      parameters_ch = Channel.fromList( some_parameters )
  
      foo( inputs_ch, parameters_ch )
  
      parameter_results = branchByElement( foo.out, some_parameters, by=1 )
      input_results = branchByElement( foo.out, some_inputs, by=0 )
  
      parameter_results[7].call().view { "Param 7: $it" }
      input_results.dna().view { "Input dna: $it" }
  }
  
  ```
  
  Results:
  
  ```
  N E X T F L O W  ~  version 20.10.0
  Launching `test.nf` [distraught_meninsky] - revision: 04df251752
  [bb/32325f] Submitted process > foo (task: 4 - prot,7)
  [a0/19fdf5] Submitted process > foo (task: 8 - dna,2)
  [e9/4c1d09] Submitted process > foo (task: 6 - dna,5)
  [a3/dcb002] Submitted process > foo (task: 2 - prot,3)
  [db/7e612e] Submitted process > foo (task: 5 - prot,1)
  [cd/dfa84e] Submitted process > foo (task: 9 - dna,7)
  [d4/6a85f2] Submitted process > foo (task: 7 - dna,3)
  [f0/8e8bf4] Submitted process > foo (task: 1 - prot,5)
  [89/f20a28] Submitted process > foo (task: 3 - prot,2)
  Input dna: [dna, 2]
  Param 7: [prot, 7]
  Input dna: [dna, 5]
  [53/056fe4] Submitted process > foo (task: 10 - dna,1)
  [f3/a61abf] Submitted process > foo (task: 12 - rna,3)
  Input dna: [dna, 7]
  [f9/a568a8] Submitted process > foo (task: 13 - rna,2)
  Param 7: [dna, 7]
  [06/72bb6c] Submitted process > foo (task: 11 - rna,5)
  Input dna: [dna, 3]
  [a3/6d80d8] Submitted process > foo (task: 14 - rna,7)
  [ac/acdb5a] Submitted process > foo (task: 15 - rna,1)
  Param 7: [rna, 7]
  Input dna: [dna, 1]
  
  ```
  
  > **Comments:**
  > **username_2**: I think `each` is even more elegant than combine, do you also have a more elegant way to split the resulting channel than `branch` ?  
  > **username_1**: @username_2 I think there is using a function and the new DSL2 - please see my update above.  
  
  ---
  
  