Answers.Body: <blockquote>
  <p>So i think there is a problem with the combination of multiple
  read-pairs as input but only one reference file?!</p>
  </blockquote>
  <p>Correct. The problem is that the <em>hisat2_index_ch</em> is a <a href="https://www.nextflow.io/docs/latest/channel.html#queue-channel" rel="nofollow noreferrer">queue channel</a>, but what you want is a <a href="https://www.nextflow.io/docs/latest/channel.html#value-channel" rel="nofollow noreferrer">value channel</a> so that the (file) values can be read an unlimited number of times without being consumed. Assuming <em>params.hisat2_index</em> is intended to supply a glob pattern, you could simply call the <a href="https://www.nextflow.io/docs/latest/operator.html#collect" rel="nofollow noreferrer">collect</a> operator to produce a list of files as the sole emission:</p>
  <pre><code>Channel
      .fromPath( params.hisat2_index )
      .collect()
      .set { hisat2_index_ch }
  </code></pre>
  <p>You would then need to update your process definition accordingly. Alternatively, have your workflow create the HISAT2 index files for you. This simplifies things considerably, by requiring only a single reference FASTA file as input. Note that value channels are implicitly created as output for a process whose inputs are all value channels. This means in the example below, <em>hisat2_index.out</em> is also a value channel:</p>
  <pre><code>params.publish_dir = './results'
  
  params.container_hisat = 'quay.io/biocontainers/hisat2:2.2.1--h87f3376_4'
  
  params.reads = '*_R{1,2}.fastq.gz'
  params.ref_fasta = 'chr1.fasta'
  
  </code></pre>
  <pre><code>process hisat2_index {
  
      container params.container_hisat
      cpus 8
  
      input:
      path fasta
  
      output:
      tuple val(fasta.baseName), path(&quot;${fasta.baseName}.*.ht2&quot;)
  
      &quot;&quot;&quot;
      hisat2-build \\
          -p <span class="math-container">${task.cpus} \\
          "$</span>{fasta}&quot; \\
          &quot;${fasta.baseName}&quot;
      &quot;&quot;&quot;
  }
  </code></pre>
  <pre><code>process hisat2_paired {
  
      tag { sample_id }
  
      container params.container_hisat
      publishDir &quot;${params.publish_dir}/hisat&quot;
  
      input:
      tuple val(sample_id), path(input_files)
      tuple val(idx), path('hisat2-idx/*')
  
      output:
      path('*.sam'), emit: hisat2_sam
      path('*.txt')
  
      script:
      def (fq1, fq2) = input_files
  
      &quot;&quot;&quot;
      hisat2 \\
        --rna-strandness RF \\
        -x &quot;hisat2-idx/<span class="math-container">${idx}" \\
        -1 $</span>{fq1} \\
        -2 <span class="math-container">${fq2} \\
        -S $</span>{sample_id}.sam \\
        &amp;&gt; ${sample_id}.hisat_summary.txt
      &quot;&quot;&quot;
  }
  </code></pre>
  <pre><code>workflow {
  
      read_pairs = Channel.fromFilePairs( params.reads )
      ref_fasta = file( params.ref_fasta )
  
      hisat2_index( ref_fasta )
  
      hisat2_paired( read_pairs, hisat2_index.out )
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T16:52:11.120Z
Answers.Text: Thank you for the help. I knew it was something simple. Simply adding the .collect() to the hisat_index_ch did the trick. But the process definition needed to be set as path or else the files are not available in the docker container.
  
  By "update the process definition" you meant changing the input from path() to val() right?
Answers.UserDisplayName: stfn.snk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-07T02:32:01.347Z
Answers.Text: @stfn.snk no, sorry, I should have elaborated. Always use the `path()` qualifier for file input/outputs. The `path()` qualifier will work for a single file or a collection of files. I just meant that you will need to update how you define your HISAT2 _INDEX_ variable. I don't think the `find` command will work as expected. It might be better to supply the basename of the index (i.e. a string) using `val()` in a `tuple()` like in the example above.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-07T02:40:17.163Z
Answers.Text: One way, could be to use the [map](https://www.nextflow.io/docs/latest/operator.html#map) operator to transform the collection, similar to how the `find` command was intended to work, .e.g `Channel.fromPath( params.hisat2_index ).collect().map { tuple( it.first().getBaseName(2), it ) }.set { ... }`
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-07T22:26:57.990Z
Answers.Text: I tried to avoid the building process, since it needs considerable computing resources (so i've read) so I used a public available HISAT Index. I think i tried the `.baseName` function already but it gave me the index filename including the number and the process failed (e.g. _genome.1.ht2_ became _genome.1_) So i ended up with the `find` solution.
  I will look further into your suggestions. Thanks again, i really appreciate your help
Answers.UserDisplayName: stfn.snk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-08T04:02:22.770Z
Answers.Text: @stfn.snk So you could use `.getBaseName(2)` to remove the double extension, or `.simpleName` if you can be sure the prefix doesn't contain a literal dot/period. Anytime!
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-06T15:42:37.613Z
Answers.LastActivityDate: 2022-11-06T16:14:45.087Z
Answers.LastEditDate: 2022-11-06T16:14:45.087Z
Answers.UserDisplayName: Steve
Body: <p>This is my first nextflow pipeline and i am struggling with the .fromFilePairs concept of Nextflow.</p>
  <p>I want to process paired-end read files and align them against a reference genome. I use Hisat2 so the index files come in form of 8 files with the suffix .ht2. Hisat needs the file path and the filename without &quot;.ht2&quot;. All worked well with a single-end version of the process. The reference also has to be passed through a channel because i use a hisat docker-container and otherwise the files are not mounted.</p>
  <p>The idea is to pass read-pairs to a preprocessing step and then pass them (as tuples, like a Channel.fromPairs()) into the hisat process and then further downstream.</p>
  <p>My problem is all file-pairs are processed during the first step but only the first pair is passed further downstream. All downstream steps finish with the one pair and the rest simply stays in the output-folder of the preprocessing step.</p>
  <p>I tried to simply skip the first step to see if the &quot;connection&quot; between the first and second process is not working but the Channel which works for the first step does not for the hisat step. So i think there is a problem with the combination of multiple read-pairs as input but only one reference file?!</p>
  <pre><code>process hisat2_paired {
    tag &quot;$sample_id&quot;
   
    container &quot;<span class="math-container">${params.container_hisat}"
    publishDir "$</span>{params.out_dir}/Nextflow_output/hisat/&quot;
  
    input:
    tuple val(sample_id), path(input_files)
    path( index_files )
  
    output:
    path('*.sam') , emit: hisat2_sam_out_ch
    path('*.txt')
  
    script:
    &quot;&quot;&quot;
    INDEX=`find -L ${index_files}/ -name &quot;*.1.ht2&quot; | sed 's/.1.ht2//'` 
    hisat2 --rna-strandness RF \
            -x \\\$INDEX \
            -1 <span class="math-container">${input_files[0]} \
        -2 $</span>{input_files[1]} \
            -S <span class="math-container">${sample_id}.sam &amp;&gt; $</span>{sample_id}.hisat_summary.txt
    &quot;&quot;&quot;
  
  </code></pre>
  <pre><code>workflow {
  
  Channel
      .fromFilePairs( &quot;${params.reads_folder}/*_{R1,R2}*f*q.gz&quot;)
      .ifEmpty { exit 1, &quot;no fastq files found at given path&quot; }
      .view()
      .set{ read_pair_ch }
  
  Channel
      .fromPath( params.hisat2_index )
      .ifEmpty { exit 1, &quot;no hisat2 index files found - path was empty&quot; }
      //.view()
      .set { hisat2_index_ch }
  
  hisat2_paired( read_pair_ch, hisat2_index_ch )
  
  }
  
  </code></pre>
  <p>I started the workflow with 3 test file-pairs and the Output on the CLI states that only one process is started although if i look at the Channel.view() all 3 file pairs are printed.</p>
  <pre><code>executor &gt;  local (1)
  [2f/30a4d6] process &gt; hisat2_paired (MC3T3-Ko1_S2_L001) [  0%] 0 of 1
  [MC3T3-Ko1_S2_L001, [/mnt/c/Path/to/MC3T3-Ko1_S2_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko1_S2_L001_R2_001.fastq.gz]]
  [MC3T3-Ko2_S4_L001, [/mnt/c/Path/to/MC3T3-Ko2_S4_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko2_S4_L001_R2_001.fastq.gz]]
  [MC3T3-Ko3_S6_L001, [/mnt/c/Path/to/MC3T3-Ko3_S6_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko3_S6_L001_R2_001.fastq.gz]]
  </code></pre>
  <p>I was already thinking about memory and resources. Since I prototype on my local PC i set the queue size in the config file to 2. I know that my hisat process needs about 7Gb RAM (for Mus.musculus) per process so there should be enough resources. Also the problem was that the local executor is not even trying to start a second process.</p>
  
Tags: <nextflow><rna-alignment>
Title: Only the first file Pair is processed from Channel.fromFilePairs()
ThreadText: # Only the first file Pair is processed from Channel.fromFilePairs()
  
  (Asked by: username_0 on 2022-11-06)
  
  This is my first nextflow pipeline and i am struggling with the .fromFilePairs concept of Nextflow.
  
  I want to process paired-end read files and align them against a reference genome. I use Hisat2 so the index files come in form of 8 files with the suffix .ht2. Hisat needs the file path and the filename without ".ht2". All worked well with a single-end version of the process. The reference also has to be passed through a channel because i use a hisat docker-container and otherwise the files are not mounted.
  
  The idea is to pass read-pairs to a preprocessing step and then pass them (as tuples, like a Channel.fromPairs()) into the hisat process and then further downstream.
  
  My problem is all file-pairs are processed during the first step but only the first pair is passed further downstream. All downstream steps finish with the one pair and the rest simply stays in the output-folder of the preprocessing step.
  
  I tried to simply skip the first step to see if the "connection" between the first and second process is not working but the Channel which works for the first step does not for the hisat step. So i think there is a problem with the combination of multiple read-pairs as input but only one reference file?!
  
  ```
  process hisat2_paired {
    tag "$sample_id"
   
    container "${params.container_hisat}"
    publishDir "${params.out_dir}/Nextflow_output/hisat/"
  
    input:
    tuple val(sample_id), path(input_files)
    path( index_files )
  
    output:
    path('*.sam') , emit: hisat2_sam_out_ch
    path('*.txt')
  
    script:
    """
    INDEX=`find -L ${index_files}/ -name "*.1.ht2" | sed 's/.1.ht2//'` 
    hisat2 --rna-strandness RF \
            -x \\\$INDEX \
            -1 ${input_files[0]} \
        -2 ${input_files[1]} \
            -S ${sample_id}.sam &> ${sample_id}.hisat_summary.txt
    """
  
  ```
  
  ```
  workflow {
  
  Channel
      .fromFilePairs( "${params.reads_folder}/*_{R1,R2}*f*q.gz")
      .ifEmpty { exit 1, "no fastq files found at given path" }
      .view()
      .set{ read_pair_ch }
  
  Channel
      .fromPath( params.hisat2_index )
      .ifEmpty { exit 1, "no hisat2 index files found - path was empty" }
      //.view()
      .set { hisat2_index_ch }
  
  hisat2_paired( read_pair_ch, hisat2_index_ch )
  
  }
  
  ```
  
  I started the workflow with 3 test file-pairs and the Output on the CLI states that only one process is started although if i look at the Channel.view() all 3 file pairs are printed.
  
  ```
  executor >  local (1)
  [2f/30a4d6] process > hisat2_paired (MC3T3-Ko1_S2_L001) [  0%] 0 of 1
  [MC3T3-Ko1_S2_L001, [/mnt/c/Path/to/MC3T3-Ko1_S2_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko1_S2_L001_R2_001.fastq.gz]]
  [MC3T3-Ko2_S4_L001, [/mnt/c/Path/to/MC3T3-Ko2_S4_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko2_S4_L001_R2_001.fastq.gz]]
  [MC3T3-Ko3_S6_L001, [/mnt/c/Path/to/MC3T3-Ko3_S6_L001_R1_001.fastq.gz, /mnt/c/Path/to/MC3T3-Ko3_S6_L001_R2_001.fastq.gz]]
  
  ```
  
  I was already thinking about memory and resources. Since I prototype on my local PC i set the queue size in the config file to 2. I know that my hisat process needs about 7Gb RAM (for Mus.musculus) per process so there should be enough resources. Also the problem was that the local executor is not even trying to start a second process.
  
  **Tags:** nextflow, rna-alignment
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-06. Score: 1 | Accepted answer
  
  > So i think there is a problem with the combination of multiple
  > read-pairs as input but only one reference file?!
  
  Correct. The problem is that the *hisat2\_index\_ch* is a [queue channel](https://www.nextflow.io/docs/latest/channel.html#queue-channel), but what you want is a [value channel](https://www.nextflow.io/docs/latest/channel.html#value-channel) so that the (file) values can be read an unlimited number of times without being consumed. Assuming *params.hisat2\_index* is intended to supply a glob pattern, you could simply call the [collect](https://www.nextflow.io/docs/latest/operator.html#collect) operator to produce a list of files as the sole emission:
  
  ```
  Channel
      .fromPath( params.hisat2_index )
      .collect()
      .set { hisat2_index_ch }
  
  ```
  
  You would then need to update your process definition accordingly. Alternatively, have your workflow create the HISAT2 index files for you. This simplifies things considerably, by requiring only a single reference FASTA file as input. Note that value channels are implicitly created as output for a process whose inputs are all value channels. This means in the example below, *hisat2\_index.out* is also a value channel:
  
  ```
  params.publish_dir = './results'
  
  params.container_hisat = 'quay.io/biocontainers/hisat2:2.2.1--h87f3376_4'
  
  params.reads = '*_R{1,2}.fastq.gz'
  params.ref_fasta = 'chr1.fasta'
  
  ```
  
  ```
  process hisat2_index {
  
      container params.container_hisat
      cpus 8
  
      input:
      path fasta
  
      output:
      tuple val(fasta.baseName), path("${fasta.baseName}.*.ht2")
  
      """
      hisat2-build \\
          -p ${task.cpus} \\
          "${fasta}" \\
          "${fasta.baseName}"
      """
  }
  
  ```
  
  ```
  process hisat2_paired {
  
      tag { sample_id }
  
      container params.container_hisat
      publishDir "${params.publish_dir}/hisat"
  
      input:
      tuple val(sample_id), path(input_files)
      tuple val(idx), path('hisat2-idx/*')
  
      output:
      path('*.sam'), emit: hisat2_sam
      path('*.txt')
  
      script:
      def (fq1, fq2) = input_files
  
      """
      hisat2 \\
        --rna-strandness RF \\
        -x "hisat2-idx/${idx}" \\
        -1 ${fq1} \\
        -2 ${fq2} \\
        -S ${sample_id}.sam \\
        &> ${sample_id}.hisat_summary.txt
      """
  }
  
  ```
  
  ```
  workflow {
  
      read_pairs = Channel.fromFilePairs( params.reads )
      ref_fasta = file( params.ref_fasta )
  
      hisat2_index( ref_fasta )
  
      hisat2_paired( read_pairs, hisat2_index.out )
  }
  
  ```
  
  > **Comments:**
  > **username_0**: Thank you for the help. I knew it was something simple. Simply adding the .collect() to the hisat_index_ch did the trick. But the process definition needed to be set as path or else the files are not available in the docker container.  By "update the process definition" you meant changing the input from path() to val() right?  
  > **username_1**: @username_0 no, sorry, I should have elaborated. Always use the `path()` qualifier for file input/outputs. The `path()` qualifier will work for a single file or a collection of files. I just meant that you will need to update how you define your HISAT2 _INDEX_ variable. I don't think the `find` command will work as expected. It might be better to supply the basename of the index (i.e. a string) using `val()` in a `tuple()` like in the example above.  
  > **username_1**: One way, could be to use the [map](https://www.nextflow.io/docs/latest/operator.html#map) operator to transform the collection, similar to how the `find` command was intended to work, .e.g `Channel.fromPath( params.hisat2_index ).collect().map { tuple( it.first().getBaseName(2), it ) }.set { ... }`  
  > **username_0**: I tried to avoid the building process, since it needs considerable computing resources (so i've read) so I used a public available HISAT Index. I think i tried the `.baseName` function already but it gave me the index filename including the number and the process failed (e.g. _genome.1.ht2_ became _genome.1_) So i ended up with the `find` solution. I will look further into your suggestions. Thanks again, i really appreciate your help  
  > **username_1**: @username_0 So you could use `.getBaseName(2)` to remove the double extension, or `.simpleName` if you can be sure the prefix doesn't contain a literal dot/period. Anytime!  
  
  ---
  
  