Answers.Body: <p>The <a href="https://www.nextflow.io/docs/latest/operator.html#merge" rel="nofollow noreferrer">merge operator</a> is deprecated and will be removed from future Nextflow releases. You'll need instead the <a href="https://www.nextflow.io/docs/latest/operator.html#join" rel="nofollow noreferrer">join operator</a> to join by chromosome. You can use the <a href="https://www.nextflow.io/docs/latest/channel.html#fromfilepairs" rel="nofollow noreferrer">fromFilePairs operator</a> to quickly get the group key, formatting it as required:</p>
  <pre><code>channel
      .fromFilePairs(&quot;/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}&quot;, size: 3)
      .map { group_key, file_list -&gt;
          tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, &quot;&quot;), file_list )
      }
      .set { ref_panel_files }
  
  
  channel
      .fromFilePairs(&quot;/some/path/genetic_map_chr*_combined_b37.txt&quot;, size: 1)
      .map { group_key, file_list -&gt;
          tuple( group_key.replaceFirst(/^genetic_map_/, &quot;&quot;), file_list.first() )
      }
      .set { genetic_map_files }
  
  
  qc1_chr
      .map { chrom, bfiles -&gt;
          tuple( chrom, bfiles.findAll { it.baseName.endsWith(&quot;step10&quot;) } )
      }
      .join( ref_panel_files )
      .join( genetic_map_files )
      .set { phasing_inputs }
  
  
  process phasing {
  
      input:
      tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs
  
      script:
      def (bed, bim, fam) = input_bed
      def (haplotypes, legend, sample) = input_ref
  
      &quot;&quot;&quot;
      shapeit2 \\
          --thread &quot;<span class="math-container">${task.cpus}" \\
          --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; \\
          --input-ref &quot;<span class="math-container">${haplotypes}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" \\
          --input-map "$</span>{input_map}&quot; \\
          --duohmm \\
          --output-max &quot;${chrom}.phased&quot; \\
          --seed 54321
      &quot;&quot;&quot;
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T16:11:10.787Z
Answers.Text: Thanks `@Steve`. Only one thing, the prefixes of `hap.gz, legend.gz and samples` are not same for each chromosome. That's why in my code there is a `*` at the beginning and another `*` at the chromosome label.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T17:56:30.347Z
Answers.Text: Hi `@Steve` I edited my question adding picture of running script and code snippet.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T22:58:20.393Z
Answers.Text: @zillurrahman: I think, though, you understand the general gist of the answer - that is, to create a tuple with a key as the first element which can then be used to join (multiple) channels. The above is just an example that uses the chromosome name as the join key. You should *not* expect a drop-in replacement for your code. I don't know what files your glob patterns match, and still don't. Your patterns could match multiple files, but I assumed you had one (hap, legend, sample) per chromosome. If the reference panel files have different prefixes, you may prefer separate channels for these.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T09:16:38.370Z
Answers.Text: Sorry for the inconveniences. I rechecked the code, there were some type. Modified them and can see joined items. But getting a new error. `No such variable: chrom` But I declared the `chrom` variable in the input.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T09:27:16.933Z
Answers.Text: The new code snippet was added to my above question.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T11:45:31.427Z
Answers.Text: @zillurrahman I can't reproduce the error you get using your new code snippet. It should have failed earlier because of this typo: `path(intput_ref)` should be `path(input_ref)`. Do you know what line the error is being produced on?
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T16:09:05.687Z
Answers.Text: Thanks `@Steve`. The error is occurring at line 20 where the output is declared.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T16:30:06.023Z
Answers.Text: In the `.nextflow.log` file, it is showing that the error is occurring at line 20. And line 20 is `tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased` where the output was declared.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T16:56:31.783Z
Answers.Text: `@Steve` you are right. There was a type in my code. The error is `Unknown variable 'input_ref`. The typo is fixed and the code is running now! Thanks a lot!
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T18:41:57.247Z
Answers.Text: If I run `phasing` separately, it works. But in combination with `prephasing`  it does not work. I mean bed|bim|fam files from channel from FilePairs works but from the output channel of previous process does not. Even I can not see the joined channel.view() in `.command.out` in the work directory. No error also. not sure what goes wrong. I added the updated code snippet in the above question.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T01:33:07.713Z
Answers.Text: @zillurrahman: I think your ouput declaration for qc1_chr needs to be: `tuple val("chr${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr`. HTH.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T07:12:51.223Z
Answers.Text: `@Steve` Thanks. I updated the output declaration of process `prephasing`. Still the process `phasing` does not work nor any error message. In the last screenshot of the above question, you can see that there is a work directory (at the left) assigned for `prephasing` process but nothing for `phasing` process. Not sure what I am doing wrong. The same `phasing` process works if I run it in a separate script where input `bed|bim|fam`  are assigned from `FilePairs`.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T10:39:34.910Z
Answers.Text: @zillurrahman: I suspect one of your input channels to join on isn't working as expected. Could you please post the output of `qc1_chr.view()`, `ref_panel.view()` and  `genetic_map.view()`? Thanks!
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T17:27:06.610Z
Answers.Text: Thanks `@Steve`. `ref_panel.view()` gives `[chr,[respective hap.gz,legend.gz,sample.gz]]`, `genetic_map.view()` gives `[chr,[respective maps]]`. The problem with filtering `qc1_chr`. `qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.view()`  shows nothing. Interestingly, same filtering method works when the `bed|bim|fam` files were from `FilePairs`. I also tried `qc1_chr.filter { it[1].name.contains('step10') }.view()` but nothing. `qc1_chr.view()` gives `[chr1-22,[respective all plink files]]`
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T23:17:19.260Z
Answers.Text: My mistake, I think you'll need to replace the filter statement with `.map { chrom, bfiles -> tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } ) }`. No filtering is required, we just need to select out the required step10 files.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T06:22:05.767Z
Answers.Text: Thank you very much `@Steve`. It works now. as there are multiple patterns, I used `channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.{hap.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(gro
  up_key.replaceAll(/^.*.hrc\.r1-1\.ega\.grch37\.|\.haplotypes\.noibd|\.haplotypes/,""),file_list)}.set {ref_panel}`. Do you think `replaceAll` is ok instead of `replaceFirst` . I got expected output using using `replaceAll`. Could I use multiple pattern using `replaceFirst`?
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T10:44:21.393Z
Answers.Text: @zillurrahman: I don't think you can since the first looks like a prefix and the last two look like suffixes. I think you'd need two replaceFirst calls. I'd be tempted to just use the first string that looks like a chromosome name: `group_key.find(~/chr\d{1,2}/)`. HTH.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T11:20:26.200Z
Answers.Text: Thanks `@Steve` I just tested the `find` option and it works. Thanks a lot! It seems fewer code and easier to write!
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-10T07:57:30.830Z
Answers.Text: Hi `@Steve`, Wondering if you get a chance to look at https://bioinformatics.stackexchange.com/questions/15698/parse-csv-text-file-to-create-arrays-in-nextflow . I just need a little help to create an array from a csv file in nextflow.
Answers.UserDisplayName: zillur rahman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T14:40:11.413Z
Answers.LastActivityDate: 2021-04-07T23:18:54.153Z
Answers.LastEditDate: 2021-04-07T23:18:54.153Z
Answers.UserDisplayName: Steve
Body: <p>I need to merge multiple channels each emits multiple file. So far I did something as below:</p>
  <pre><code>process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
  &quot;&quot;&quot;
  PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
  &quot;&quot;&quot;
  }
  channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
  mp_map}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
  imp_ref}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
  hape_legend;imp_legend}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
  pe_sample;imp_sample}
  qc1_chr.filter{name, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.into {shape_in;imp_in}
  
  shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
      
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  each file(penta) from shape_para
  output:
  path(&quot;*step10*&quot;) into phased
  script:
  def (in, map, ref, legend, sample) = penta
  &quot;&quot;&quot;
  shapeit2 -T 32 --input-bed &quot;<span class="math-container">${in}" --input-map "$</span>{map}&quot; --input-ref &quot;<span class="math-container">${ref}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" --duohmm -O "$</span>{in}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p>The &quot;prephasing&quot; process works fine and produce expected output. But the &quot;phasing&quot; process does not produce any output nor any error. Any help?</p>
  <p>I have tried according to <code>@Steve</code> as below:</p>
  <pre><code>channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt; tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  shapeit2 --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p>But getting no output so far, (attached image<a href="https://i.sstatic.net/B1PtL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B1PtL.png" alt="phasing process" /></a>)</p>
  <p>I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file_list]]. After merging I can see [chr1-22,[joined_file_list].</p>
  <pre><code>channel
      .fromFilePairs( &quot;output3/*.{bed,bim,fam}&quot;, size:3 )
      .set { plink_data }
  plink_data.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.
  replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.set {in_pl}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
  group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  //genetic_map.view()
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
  p.gz,legend.gz,samples}',size:3).map {group_key,file_list -&gt; tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {re
  f_panel}
  //ref_panel.view()
  
  
  
  in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  process phasing {
  //publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) in
  to phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  //shapeit=&quot;/cluster/projects/p697/projects/moba_qc_imputation/software/shap
  eit&quot;
  shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{samp
  le}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p><a href="https://i.sstatic.net/CR2sW.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CR2sW.jpg" alt="joined channel example view" /></a>
  But now I get error:</p>
  <pre><code>No such variable: chrom
  </code></pre>
  <p>But I decalred &quot;chrom&quot; in the input</p>
  <p>The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run <code>phasing</code> solo and all the inputs from channel from FilePairs it works. But if I run <code>prephasing</code> and <code>phasing</code> together, the <code>prephasing</code> process works but for <code>phasing</code> process not output and no error; Not sure what I am doing wrong.</p>
  <pre><code>hrc_sites_file = file(params.hrc_sites_file)
  
  Channel.fromFilePairs( &quot;${params.input_dir}/*.{bed,bim,fam}&quot;, size:3 ).set { plink_data }
  
  Channel.of(1..22).set { chroms }
  params.out='output/'
  
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*log&quot;) into qc1_log
  tuple val(&quot;<span class="math-container">${chrom}"),path("*-chr$</span>{chrom}.step4-HRC.txt&quot;) into qc1_hrc
  path(&quot;chr<span class="math-container">${chrom}.step1b.snplist.txt") into qc1_snps
  path("chr$</span>{chrom}.step1a.dupvar&quot;) into qc1_dups
  &quot;&quot;&quot;
  PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
  &quot;&quot;&quot;
  }
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*gen
  etic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt;
   tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.join(ref_panel).joi
  n(genetic_map).into {shape_in;imp_in}
  imp_in.view()
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.*&quot;) into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.step10&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p><a href="https://i.sstatic.net/bjxQJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bjxQJ.png" alt="prephasing works but phasing does not" /></a></p>
  
Tags: <nextflow>
Title: Merge multiple channels each emits multiple files in nextflow
ThreadText: # Merge multiple channels each emits multiple files in nextflow
  
  (Asked by: username_0 on 2021-04-05)
  
  I need to merge multiple channels each emits multiple file. So far I did something as below:
  
  ```
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
  """
  PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
  """
  }
  channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
  mp_map}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
  imp_ref}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
  hape_legend;imp_legend}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
  pe_sample;imp_sample}
  qc1_chr.filter{name, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.into {shape_in;imp_in}
  
  shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
      
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  each file(penta) from shape_para
  output:
  path("*step10*") into phased
  script:
  def (in, map, ref, legend, sample) = penta
  """
  shapeit2 -T 32 --input-bed "${in}" --input-map "${map}" --input-ref "${ref}" "${legend}" "${sample}" --duohmm -O "${in}.phased" --seed 54321
  """
  }
  
  ```
  
  The "prephasing" process works fine and produce expected output. But the "phasing" process does not produce any output nor any error. Any help?
  
  I have tried according to `@username_1` as below:
  
  ```
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^genetic_map_/,""),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -> tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  shapeit2 --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.phased" --seed 54321
  """
  }
  
  ```
  
  But getting no output so far, (attached image[![phasing process](https://i.sstatic.net/B1PtL.png)](https://i.sstatic.net/B1PtL.png))
  
  I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file\_list]]. After merging I can see [chr1-22,[joined\_file\_list].
  
  ```
  channel
      .fromFilePairs( "output3/*.{bed,bim,fam}", size:3 )
      .set { plink_data }
  plink_data.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.
  replaceFirst(/^*\.step10/,""), file_list)}.set {in_pl}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
  group_key, file_list -> tuple(group_key.replaceFirst(/^*genetic_map_/,""),file_list.first())}.set {genetic_map}
  //genetic_map.view()
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
  p.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {re
  f_panel}
  //ref_panel.view()
  
  in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  process phasing {
  //publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") in
  to phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  //shapeit="/cluster/projects/p697/projects/moba_qc_imputation/software/shap
  eit"
  shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${samp
  le}" --duohmm --output-max "${chrom}.phased" --seed 54321
  """
  }
  
  ```
  
  [![joined channel example view](https://i.sstatic.net/CR2sW.jpg)](https://i.sstatic.net/CR2sW.jpg)
  But now I get error:
  
  ```
  No such variable: chrom
  
  ```
  
  But I decalred "chrom" in the input
  
  The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run `phasing` solo and all the inputs from channel from FilePairs it works. But if I run `prephasing` and `phasing` together, the `prephasing` process works but for `phasing` process not output and no error; Not sure what I am doing wrong.
  
  ```
  hrc_sites_file = file(params.hrc_sites_file)
  
  Channel.fromFilePairs( "${params.input_dir}/*.{bed,bim,fam}", size:3 ).set { plink_data }
  
  Channel.of(1..22).set { chroms }
  params.out='output/'
  
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
  tuple val("${chrom}"),path("chr${chrom}*log") into qc1_log
  tuple val("${chrom}"),path("*-chr${chrom}.step4-HRC.txt") into qc1_hrc
  path("chr${chrom}.step1b.snplist.txt") into qc1_snps
  path("chr${chrom}.step1a.dupvar") into qc1_dups
  """
  PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
  """
  }
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^*gen
  etic_map_/,""),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list ->
   tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.replaceFirst(/^*\.step10/,""), file_list)}.join(ref_panel).joi
  n(genetic_map).into {shape_in;imp_in}
  imp_in.view()
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.*") into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.step10" --seed 54321
  """
  }
  
  ```
  
  [![prephasing works but phasing does not](https://i.sstatic.net/bjxQJ.png)](https://i.sstatic.net/bjxQJ.png)
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-05. Score: 3 | Accepted answer
  
  The [merge operator](https://www.nextflow.io/docs/latest/operator.html#merge) is deprecated and will be removed from future Nextflow releases. You'll need instead the [join operator](https://www.nextflow.io/docs/latest/operator.html#join) to join by chromosome. You can use the [fromFilePairs operator](https://www.nextflow.io/docs/latest/channel.html#fromfilepairs) to quickly get the group key, formatting it as required:
  
  ```
  channel
      .fromFilePairs("/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}", size: 3)
      .map { group_key, file_list ->
          tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, ""), file_list )
      }
      .set { ref_panel_files }
  
  channel
      .fromFilePairs("/some/path/genetic_map_chr*_combined_b37.txt", size: 1)
      .map { group_key, file_list ->
          tuple( group_key.replaceFirst(/^genetic_map_/, ""), file_list.first() )
      }
      .set { genetic_map_files }
  
  qc1_chr
      .map { chrom, bfiles ->
          tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } )
      }
      .join( ref_panel_files )
      .join( genetic_map_files )
      .set { phasing_inputs }
  
  process phasing {
  
      input:
      tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs
  
      script:
      def (bed, bim, fam) = input_bed
      def (haplotypes, legend, sample) = input_ref
  
      """
      shapeit2 \\
          --thread "${task.cpus}" \\
          --input-bed "${bed}" "${bim}" "${fam}" \\
          --input-ref "${haplotypes}" "${legend}" "${sample}" \\
          --input-map "${input_map}" \\
          --duohmm \\
          --output-max "${chrom}.phased" \\
          --seed 54321
      """
  }
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks `@username_1`. Only one thing, the prefixes of `hap.gz, legend.gz and samples` are not same for each chromosome. That's why in my code there is a `*` at the beginning and another `*` at the chromosome label.  
  > **username_0**: Hi `@username_1` I edited my question adding picture of running script and code snippet.  
  > **username_1**: @username_0: I think, though, you understand the general gist of the answer - that is, to create a tuple with a key as the first element which can then be used to join (multiple) channels. The above is just an example that uses the chromosome name as the join key. You should *not* expect a drop-in replacement for your code. I don't know what files your glob patterns match, and still don't. Your patterns could match multiple files, but I assumed you had one (hap, legend, sample) per chromosome. If the reference panel files have different prefixes, you may prefer separate channels for these.  
  > **username_0**: Sorry for the inconveniences. I rechecked the code, there were some type. Modified them and can see joined items. But getting a new error. `No such variable: chrom` But I declared the `chrom` variable in the input.  
  > **username_0**: The new code snippet was added to my above question.  
  > **username_1**: @username_0 I can't reproduce the error you get using your new code snippet. It should have failed earlier because of this typo: `path(intput_ref)` should be `path(input_ref)`. Do you know what line the error is being produced on?  
  > **username_0**: Thanks `@username_1`. The error is occurring at line 20 where the output is declared.  
  > **username_0**: In the `.nextflow.log` file, it is showing that the error is occurring at line 20. And line 20 is `tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased` where the output was declared.  
  > **username_0**: `@username_1` you are right. There was a type in my code. The error is `Unknown variable 'input_ref`. The typo is fixed and the code is running now! Thanks a lot!  
  > **username_0**: If I run `phasing` separately, it works. But in combination with `prephasing`  it does not work. I mean bed|bim|fam files from channel from FilePairs works but from the output channel of previous process does not. Even I can not see the joined channel.view() in `.command.out` in the work directory. No error also. not sure what goes wrong. I added the updated code snippet in the above question.  
  > **username_1**: @username_0: I think your ouput declaration for qc1_chr needs to be: `tuple val("chr${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr`. HTH.  
  > **username_0**: `@username_1` Thanks. I updated the output declaration of process `prephasing`. Still the process `phasing` does not work nor any error message. In the last screenshot of the above question, you can see that there is a work directory (at the left) assigned for `prephasing` process but nothing for `phasing` process. Not sure what I am doing wrong. The same `phasing` process works if I run it in a separate script where input `bed|bim|fam`  are assigned from `FilePairs`.  
  > **username_1**: @username_0: I suspect one of your input channels to join on isn't working as expected. Could you please post the output of `qc1_chr.view()`, `ref_panel.view()` and  `genetic_map.view()`? Thanks!  
  > **username_0**: Thanks `@username_1`. `ref_panel.view()` gives `[chr,[respective hap.gz,legend.gz,sample.gz]]`, `genetic_map.view()` gives `[chr,[respective maps]]`. The problem with filtering `qc1_chr`. `qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.view()`  shows nothing. Interestingly, same filtering method works when the `bed|bim|fam` files were from `FilePairs`. I also tried `qc1_chr.filter { it[1].name.contains('step10') }.view()` but nothing. `qc1_chr.view()` gives `[chr1-22,[respective all plink files]]`  
  > **username_1**: My mistake, I think you'll need to replace the filter statement with `.map { chrom, bfiles -> tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } ) }`. No filtering is required, we just need to select out the required step10 files.  
  > **username_0**: Thank you very much `@username_1`. It works now. as there are multiple patterns, I used `channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.{hap.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(gro up_key.replaceAll(/^.*.hrc\.r1-1\.ega\.grch37\.|\.haplotypes\.noibd|\.haplotypes/,""),file_list)}.set {ref_panel}`. Do you think `replaceAll` is ok instead of `replaceFirst` . I got expected output using using `replaceAll`. Could I use multiple pattern using `replaceFirst`?  
  > **username_1**: @username_0: I don't think you can since the first looks like a prefix and the last two look like suffixes. I think you'd need two replaceFirst calls. I'd be tempted to just use the first string that looks like a chromosome name: `group_key.find(~/chr\d{1,2}/)`. HTH.  
  > **username_0**: Thanks `@username_1` I just tested the `find` option and it works. Thanks a lot! It seems fewer code and easier to write!  
  > **username_0**: Hi `@username_1`, Wondering if you get a chance to look at https://bioinformatics.stackexchange.com/questions/15698/parse-csv-text-file-to-create-arrays-in-nextflow . I just need a little help to create an array from a csv file in nextflow.  
  
  ---
  
  