Answers.Body: <p>I am reluctant to go through this wall of code, but generally it is advisable to always keep bam files and its index in a tuple so the index is staged with its bam file.</p>
  <p>For example a process that creates the bam and now emits a tuple with bam and bai file:</p>
  <pre><code>(...)
  output:
  tuple path(&quot;<span class="math-container">${id}.bam"), path("$</span>{id}.bam.bai&quot;), emit: tuple_bam
  </code></pre>
  <p>and then a process that receives such a <code>tuple_bam</code>:</p>
  <pre><code>(...)
  input:
  tuple path(bam), path(bai)
  </code></pre>
  <p>The first entry is the bam, the second the index.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T13:46:24.227Z
Answers.Text: Thanks it worked.  I cannot understand why only one chromosome is being processed in lancet despite a channel being passed.
Answers.UserDisplayName: Death Metal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T07:08:48.877Z
Answers.LastActivityDate: 2023-07-18T07:08:48.877Z
Answers.UserDisplayName: ATpoint
Body: <p>I've a pipeline with many processes.<br />
  One of the steps is to run lancet tool. Lancet tool runs per chromosome (1-22, x,y). The input in the lancet step is output from the applybqsr (gatk).</p>
  <p>There are tumor and normal bam generated along with the index. human genome FASTA has a index. When I run lancet through the main.nf I run into an error</p>
  <blockquote>
  <p>ERROR: index not found for BAM file</p>
  </blockquote>
  <p>I've a config file as:</p>
  <pre><code>params {
  
          hg38genome=&quot;/sc/arion/reference_genome/human/hg38/Homo_sapiens_assembly38.fasta.gz&quot;
          outdir = '/sc/arion/output_pipeline_nextflow/test_tiny_datasets/'
          dbsnp=&quot;/sc/arion/projects/Daphni2_Pipeline_resources/resourceDBsnp146/dbsnp_146.hg38.vcf.gz&quot;
          target=&quot;/sc/arion/projects/Twist_Exome_Target_hg38.padded.interval_list.bed/Twist_Exome_Target_hg38.padded.interval_list.bed&quot;
  
  }
          withName: applybqsr {
          publishDir = [
                  [  
                  path: &quot;<span class="math-container">${params.outdir}/applybqsr/normal/",
                  mode: 'copy',
                  pattern:'*normal*'
                  ],
                  [
                  path: "$</span>{params.outdir}/applybqsr/tumor/&quot;,
                  mode: 'copy',
                  pattern:'*tumor*'
                  ]       
          ]
          }
          //applybqsr ends        
  
          
          withName: lancet {
          publishDir = [  
                  path: &quot;${params.outdir}/lancet/&quot;,
                  mode: 'copy'
          ]
          }
          //lancet ends
  </code></pre>
  <p>I have applybqsr as:</p>
  <pre><code>process applybqsr{
  
          maxForks 3
          debug true
  
          memory '6 GB'
  
          input:
          tuple val(sample_id_tumor), path(markbam_tumor)
          tuple val(sample_id_tumor), path(recaltable_tumor)
          tuple val(sample_id_normal), path(markbam_normal)
          tuple val(sample_id_normal), path(recaltable_normal)
  
          output:
          tuple val(sample_id_tumor), path(&quot;<span class="math-container">${sample_id_tumor}.sorted.markdup.recal.bam"), emit: recal_bam_tumor
          tuple val(sample_id_tumor), path("$</span>{sample_id_tumor}.sorted.markdup.recal.bai&quot;),  emit: recal_bai_tumor
          tuple val(sample_id_normal), path(&quot;<span class="math-container">${sample_id_normal}.sorted.markdup.recal.bam"), emit: recal_bam_normal
          tuple val(sample_id_normal), path("$</span>{sample_id_normal}.sorted.markdup.recal.bai&quot;),  emit: recal_bai_normal
          
          script:
          &quot;&quot;&quot;
          ml gatk/4.1.3.0 samtools/1.9
          samtools index <span class="math-container">${markbam_tumor}
          samtools index $</span>{markbam_normal}
  
          gatk ApplyBQSR \\
          -R <span class="math-container">$params.hg38genome \\
          -I $</span>{markbam_tumor} \\
          --bqsr-recal-file <span class="math-container">${recaltable_tumor} \\
          -O $</span>{sample_id_tumor}.sorted.markdup.recal.bam \\
          -L <span class="math-container">$params.target \\
          --create-output-bam-index true \\
          --java-options -Xmx$</span>{task.memory.toGiga()}g
  
          gatk ApplyBQSR \\
          -R <span class="math-container">$params.hg38genome \\
          -I $</span>{markbam_normal} \\
          --bqsr-recal-file <span class="math-container">${recaltable_normal} \\
          -O $</span>{sample_id_normal}.sorted.markdup.recal.bam \\
          -L <span class="math-container">$params.target \\
          --create-output-bam-index true \\
          --java-options -Xmx$</span>{task.memory.toGiga()}g
  
  
          &quot;&quot;&quot;
  
  }
  workflow.onComplete { 
          log.info ( workflow.success ? &quot;Done applyBQSR!&quot; : &quot;Oops .. something went wrong in applyBQSR&quot; )
  }
  </code></pre>
  <p>I've lancet.nf as:</p>
  <pre><code>process lancet {
          maxForks 3
          debug true
  
          input:
          tuple val(sample_id_tumor),path(bqsrbam_tumor)
          tuple val(sample_id_normal),path(bqsrbam_normal)
          val chromosome
  
          output:
          path &quot;CHR<span class="math-container">${chromosome}_$</span>{sample_name}.lancet.vcf&quot;
  
          script:
  
          sample_name=sample_id_normal.split('_normal')[0]
  
  &quot;&quot;&quot;
  ml lancet
  
  lancet --tumor <span class="math-container">${bqsrbam_tumor} --normal $</span>{bqsrbam_normal} --ref <span class="math-container">$params.hg38genome --reg "chr"$</span>chromosome --num-threads 8 &gt; &quot;CHR&quot;<span class="math-container">${chromosome}"_$</span>{sample_name}.lancet.vcf&quot; 
  &quot;&quot;&quot;
  
  }
  
  workflow.onComplete {
            log.info(workflow.success ? &quot;Done lancet!&quot; : &quot;Oops .. something went wrong in lancet&quot; )
   }
  </code></pre>
  <p>I have main.nf as:</p>
  <pre><code>params.outdir_index=&quot;/sc/arion/output_pipeline_nextflow/test_tiny_datasets/bwa_index&quot;
  params.raw_files_normal=&quot;/sc/arion/tiny/all_tumors_normals/*_normal_*{R1,R2}.fastq&quot;
  params.raw_files_tumor=&quot;/sc/arion/tiny/all_tumors_normals/*_tumor_*{R1,R2}.fastq&quot;
  
  include { FASTP} from './fastp_process.nf'
  include {bwa_index} from './index_process.nf'
  
  include { align_bwa_mem} from './bwamem_process_already_index.nf'
  include { gatk_markduplicates} from './gatk_markduplicates_process.nf'
  include {setupnmdtags} from './setupnmdtags_process.nf'
  include { recalibrate_bam } from './recalibratebam_process.nf'
  include { applybqsr } from './applybqsr_process.nf'
  include { mutect2 } from './mutect2_process.nf'
  include { lancet } from './lancet_process.nf'
  
  workflow {
  
          normal_read_pairs=Channel.fromFilePairs(params.raw_files_normal)
          tumor_read_pairs=Channel.fromFilePairs(params.raw_files_tumor)
          //FASTP   align_bwa_mem        gatk_markduplicates         setupnmdtags        recalibrate_bam 
          
          applybqsr(gatk_markduplicates.out.sorted_bam_mark_duplicates_tumor,recalibrate_bam.out.recalib_table_tumor,gatk_markduplicates.out.sorted_bam_mark_duplicates_normal,recalibrate_bam.out.recalib_table_normal)
  
          mutect2(applybqsr.out.recal_bam_tumor,applybqsr.out.recal_bam_normal)
          
          chr=Channel.of( 1..22, 'X', 'Y' )
          lancet(applybqsr.out.recal_bam_tumor,applybqsr.out.recal_bam_normal,chr)
  }
  </code></pre>
  <p>Lancet runs fine if I run it hard coded into a process code block with same input (tumor-normal bam) and human genome FASTA. How do I enable index files to be staged into the lancet process generated one step before?</p>
  
Tags: <nextflow>
Title: nextflow: unable to find index
ThreadText: # nextflow: unable to find index
  
  (Asked by: username_0 on 2023-07-17)
  
  I've a pipeline with many processes.  
  One of the steps is to run lancet tool. Lancet tool runs per chromosome (1-22, x,y). The input in the lancet step is output from the applybqsr (gatk).
  
  There are tumor and normal bam generated along with the index. human genome FASTA has a index. When I run lancet through the main.nf I run into an error
  
  > ERROR: index not found for BAM file
  
  I've a config file as:
  
  ```
  params {
  
          hg38genome="/sc/arion/reference_genome/human/hg38/Homo_sapiens_assembly38.fasta.gz"
          outdir = '/sc/arion/output_pipeline_nextflow/test_tiny_datasets/'
          dbsnp="/sc/arion/projects/Daphni2_Pipeline_resources/resourceDBsnp146/dbsnp_146.hg38.vcf.gz"
          target="/sc/arion/projects/Twist_Exome_Target_hg38.padded.interval_list.bed/Twist_Exome_Target_hg38.padded.interval_list.bed"
  
  }
          withName: applybqsr {
          publishDir = [
                  [  
                  path: "${params.outdir}/applybqsr/normal/",
                  mode: 'copy',
                  pattern:'*normal*'
                  ],
                  [
                  path: "${params.outdir}/applybqsr/tumor/",
                  mode: 'copy',
                  pattern:'*tumor*'
                  ]       
          ]
          }
          //applybqsr ends        
  
          
          withName: lancet {
          publishDir = [  
                  path: "${params.outdir}/lancet/",
                  mode: 'copy'
          ]
          }
          //lancet ends
  
  ```
  
  I have applybqsr as:
  
  ```
  process applybqsr{
  
          maxForks 3
          debug true
  
          memory '6 GB'
  
          input:
          tuple val(sample_id_tumor), path(markbam_tumor)
          tuple val(sample_id_tumor), path(recaltable_tumor)
          tuple val(sample_id_normal), path(markbam_normal)
          tuple val(sample_id_normal), path(recaltable_normal)
  
          output:
          tuple val(sample_id_tumor), path("${sample_id_tumor}.sorted.markdup.recal.bam"), emit: recal_bam_tumor
          tuple val(sample_id_tumor), path("${sample_id_tumor}.sorted.markdup.recal.bai"),  emit: recal_bai_tumor
          tuple val(sample_id_normal), path("${sample_id_normal}.sorted.markdup.recal.bam"), emit: recal_bam_normal
          tuple val(sample_id_normal), path("${sample_id_normal}.sorted.markdup.recal.bai"),  emit: recal_bai_normal
          
          script:
          """
          ml gatk/4.1.3.0 samtools/1.9
          samtools index ${markbam_tumor}
          samtools index ${markbam_normal}
  
          gatk ApplyBQSR \\
          -R $params.hg38genome \\
          -I ${markbam_tumor} \\
          --bqsr-recal-file ${recaltable_tumor} \\
          -O ${sample_id_tumor}.sorted.markdup.recal.bam \\
          -L $params.target \\
          --create-output-bam-index true \\
          --java-options -Xmx${task.memory.toGiga()}g
  
          gatk ApplyBQSR \\
          -R $params.hg38genome \\
          -I ${markbam_normal} \\
          --bqsr-recal-file ${recaltable_normal} \\
          -O ${sample_id_normal}.sorted.markdup.recal.bam \\
          -L $params.target \\
          --create-output-bam-index true \\
          --java-options -Xmx${task.memory.toGiga()}g
  
          """
  
  }
  workflow.onComplete { 
          log.info ( workflow.success ? "Done applyBQSR!" : "Oops .. something went wrong in applyBQSR" )
  }
  
  ```
  
  I've lancet.nf as:
  
  ```
  process lancet {
          maxForks 3
          debug true
  
          input:
          tuple val(sample_id_tumor),path(bqsrbam_tumor)
          tuple val(sample_id_normal),path(bqsrbam_normal)
          val chromosome
  
          output:
          path "CHR${chromosome}_${sample_name}.lancet.vcf"
  
          script:
  
          sample_name=sample_id_normal.split('_normal')[0]
  
  """
  ml lancet
  
  lancet --tumor ${bqsrbam_tumor} --normal ${bqsrbam_normal} --ref $params.hg38genome --reg "chr"$chromosome --num-threads 8 > "CHR"${chromosome}"_${sample_name}.lancet.vcf" 
  """
  
  }
  
  workflow.onComplete {
            log.info(workflow.success ? "Done lancet!" : "Oops .. something went wrong in lancet" )
   }
  
  ```
  
  I have main.nf as:
  
  ```
  params.outdir_index="/sc/arion/output_pipeline_nextflow/test_tiny_datasets/bwa_index"
  params.raw_files_normal="/sc/arion/tiny/all_tumors_normals/*_normal_*{R1,R2}.fastq"
  params.raw_files_tumor="/sc/arion/tiny/all_tumors_normals/*_tumor_*{R1,R2}.fastq"
  
  include { FASTP} from './fastp_process.nf'
  include {bwa_index} from './index_process.nf'
  
  include { align_bwa_mem} from './bwamem_process_already_index.nf'
  include { gatk_markduplicates} from './gatk_markduplicates_process.nf'
  include {setupnmdtags} from './setupnmdtags_process.nf'
  include { recalibrate_bam } from './recalibratebam_process.nf'
  include { applybqsr } from './applybqsr_process.nf'
  include { mutect2 } from './mutect2_process.nf'
  include { lancet } from './lancet_process.nf'
  
  workflow {
  
          normal_read_pairs=Channel.fromFilePairs(params.raw_files_normal)
          tumor_read_pairs=Channel.fromFilePairs(params.raw_files_tumor)
          //FASTP   align_bwa_mem        gatk_markduplicates         setupnmdtags        recalibrate_bam 
          
          applybqsr(gatk_markduplicates.out.sorted_bam_mark_duplicates_tumor,recalibrate_bam.out.recalib_table_tumor,gatk_markduplicates.out.sorted_bam_mark_duplicates_normal,recalibrate_bam.out.recalib_table_normal)
  
          mutect2(applybqsr.out.recal_bam_tumor,applybqsr.out.recal_bam_normal)
          
          chr=Channel.of( 1..22, 'X', 'Y' )
          lancet(applybqsr.out.recal_bam_tumor,applybqsr.out.recal_bam_normal,chr)
  }
  
  ```
  
  Lancet runs fine if I run it hard coded into a process code block with same input (tumor-normal bam) and human genome FASTA. How do I enable index files to be staged into the lancet process generated one step before?
  
  **Tags:** nextflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-18. Score: 2 | Accepted answer
  
  I am reluctant to go through this wall of code, but generally it is advisable to always keep bam files and its index in a tuple so the index is staged with its bam file.
  
  For example a process that creates the bam and now emits a tuple with bam and bai file:
  
  ```
  (...)
  output:
  tuple path("${id}.bam"), path("${id}.bam.bai"), emit: tuple_bam
  
  ```
  
  and then a process that receives such a `tuple_bam`:
  
  ```
  (...)
  input:
  tuple path(bam), path(bai)
  
  ```
  
  The first entry is the bam, the second the index.
  
  > **Comments:**
  > **username_0**: Thanks it worked.  I cannot understand why only one chromosome is being processed in lancet despite a channel being passed.  
  
  ---
  
  