Answers.Body: <p>Your <a href="https://www.nextflow.io/docs/stable/cache-and-resume.html#task-cache" rel="nofollow noreferrer">task cache</a> is being invalidated because your <a href="https://www.nextflow.io/docs/stable/cache-and-resume.html#non-deterministic-process-inputs" rel="nofollow noreferrer">process inputs are non-deterministic</a>. Like in the example, the solution is to <a href="https://www.nextflow.io/docs/stable/operator.html#join" rel="nofollow noreferrer">join</a> the two output channels before process invocation. Another way is to simply output the trimmed reads in a single channel, for example:</p>
  <pre><code>process cut_adapt_trim {
  
      tag &quot;$replicateId&quot;
  
      publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
      conda 'cutadapt'
  
      input:
      tuple val(replicateId), path(reads)
      
      output:
      tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), path("$</span>{replicateId}.2.trimmed.fastq&quot;)
      
      script:
      &quot;&quot;&quot;
      cutadapt \\
          -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG \\
          -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG \\
          -o &quot;<span class="math-container">${replicateId}.1.trimmed.fastq" \\
          -p "$</span>{replicateId}.2.trimmed.fastq&quot; \\
          ${reads}
      &quot;&quot;&quot;
  }
  </code></pre>
  <pre><code>process align_to_genome {
  
      tag &quot;$replicateId&quot;
  
      publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
      conda 'bowtie2'
  
      input:
      tuple val(replicateId), path(fastq_1), path(fastq_2)
      path index
  
      output:
      tuple val(replicateId), path(&quot;${replicateId}.bam&quot;)
  
      script:
      def bwt_index = index[0].simpleName
  
      &quot;&quot;&quot;
      bowtie2 \\
          -p 64 \\
          -q \\
          -x &quot;<span class="math-container">${bwt_index}" \\
          -1 "$</span>{fastq_1}&quot; \\
          -2 &quot;<span class="math-container">${fastq_2}" |
      samtools view \\
          -b \\
          -o "$</span>{replicateId}.bam&quot; \\
          -
      &quot;&quot;&quot;
  }
  </code></pre>
  <pre><code>workflow{
  
      reads_ch = Channel.fromFilePairs( params.reads )
      index_ch = Channel.fromPath( params.index )
  
      cut_adapt_trim( reads_ch )
  
      align_to_genome( cut_adapt_trim.out, index_ch )
  
      align_to_genome.out.view()
  
  }
  </code></pre>
  <p>Run using:</p>
  <pre><code>nextflow run run_nextflow_ATACseq.nf -with-conda -resume 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-06T13:35:59.747Z
Answers.LastActivityDate: 2024-10-06T13:45:56.897Z
Answers.LastEditDate: 2024-10-06T13:45:56.897Z
Answers.UserDisplayName: Steve
Answers.Body: <p>You can achieve your desired checkpointing with some if statements and a user defined flag pretty easily. Something like:</p>
  <pre><code>#!/usr/bin/env nextflow
  nextflow.enable.dsl = 2
  
  params.skip_trim = null 
  params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
  params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
  params.trimmed_dir='01_trimmed'
  
  process cut_adapt_trim {
     publishDir ${params.trimmed_dir}, mode: 'copy'
  
     input:
     tuple val(replicateId), path(reads)
  
     output:
     tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
  tuple val(replicateId), path("$</span>{replicateId}.2.trimmed.fastq&quot;), emit: trimmed_pair_2
  
     script:
     &quot;&quot;&quot;
     cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o <span class="math-container">${replicateId}.1.trimmed.fastq -p $</span>{replicateId}.2.trimmed.fastq ${reads}
     &quot;&quot;&quot;
  }
  
  
  process align_to_genome {
     publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
     input:
     path index
     tuple val(replicateId), path(&quot;/Volumes/FrailtData/ATAC_Seq_downloaded/results/<span class="math-container">${replicateId}.1.trimmed.fastq")
  tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/$</span>{replicateId}.2.trimmed.fastq&quot;)
  
     output:
     tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.sam"), emit:aligned_sam
  tuple val(replicateId), path("$</span>{replicateId}.bam&quot;), emit:aligned_bam
  
     script:
     &quot;&quot;&quot;
     def bwt_index = index[0].simpleName
  
  
     /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x <span class="math-container">${bwt_index} -1 $</span>{replicateId}.1.trimmed.fastq -2 <span class="math-container">${replicateId}.2.trimmed.fastq -S $</span>{replicateId}.Aligned.sam 2&gt; <span class="math-container">${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - &gt; $</span>{replicateId}.bam
  
     &quot;&quot;&quot;
  }
  
  
  
  workflow{
     index_ch= Channel.fromPath(params.index)
     if( params.skip_trim == false ) {
         reads_ch= Channel.fromFilePairs(params.reads)
         cut_adapt_trim(reads_ch)
         align_to_genome(index_ch,cut_adapt_trim.out)
     } else if( params.skip_trim ) {
         trim_ch= Channel.fromFilePairs(&quot;${params.trimmed_dir}/*_{1,2}.trimmed.fastq&quot;)
         align_to_genome(index_ch,trim_ch)
     }
  }
  </code></pre>
  <p>Then to execute the skip you would use:</p>
  <pre><code>nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results --skip_trim
  </code></pre>
  <p>Play around with the <code>trimmed_dir</code> variable, I don't know what you are mounting to. But using variables for a lot of the paths and options into the params section makes it easier to adjust in the future for other files.</p>
  <p>There is another point I would like to address. Nextflow emits and inputs files in random order. So by splitting the R1 and R2 reads into separate channels, you are going to have a lot of mixed samples as your output. Make the output a single tuple: <code>tuple val(id), path(r1), path(r2), emit: trimmed_reads</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-07T20:48:28.443Z
Answers.Text: Thank you again for the feedback. Very helpful!
Answers.UserDisplayName: siddhartha das
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-06T10:36:07.337Z
Answers.LastActivityDate: 2024-10-06T10:36:07.337Z
Answers.UserDisplayName: dthorbur
Body: <p>I am trying to use nextflow to process some ATAC seq data. The data is paired end and the files are huge. So, I first trimmed the adapters and now I want to align them to the genome.
  My script is shown below. Now, I want to align them using bowtie2. However, when I run the script, the trimming is being rerun again instead of starting at the alignment part.
  Below is my call to nextflow at the command line</p>
  <pre><code>nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results -resume
  </code></pre>
  <p>I was wondering what I am doing wrong such that the resume flag is not working?</p>
  <p>Thanks for any suggestions!</p>
  <pre><code>#!/usr/bin/env nextflow
  
  nextflow.enable.dsl = 2
  
  
  params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
  params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
  
  
  
  /*
  * Trim atac seq reads to align to genome with cutadapt
  */
  
  
  
  process cut_adapt_trim {
  
  
  publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
  input:
  
  tuple val(replicateId), path(reads)
  
  
  
  output:
  
  tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
  tuple val(replicateId), path("$</span>{replicateId}.2.trimmed.fastq&quot;), emit: trimmed_pair_2
  
  
  
  script:
  &quot;&quot;&quot;
  
  
  cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o <span class="math-container">${replicateId}.1.trimmed.fastq -p $</span>{replicateId}.2.trimmed.fastq ${reads}
  
  &quot;&quot;&quot;
  
  
  }
  
  
  /*
  * Align trimmed atac seq reads to genome
  */
  
  
  process align_to_genome {
  
  publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
  
  input:
  
  path index
  
  tuple val(replicateId), path(&quot;/Volumes/FrailtData/ATAC_Seq_downloaded/results/<span class="math-container">${replicateId}.1.trimmed.fastq")
  tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/$</span>{replicateId}.2.trimmed.fastq&quot;)
  
  
  output:
  
  tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.sam"), emit:aligned_sam
  tuple val(replicateId), path("$</span>{replicateId}.bam&quot;), emit:aligned_bam
  
  
  script:
  
  &quot;&quot;&quot;
  def bwt_index = index[0].simpleName
  
  
  /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x <span class="math-container">${bwt_index} -1 $</span>{replicateId}.1.trimmed.fastq -2 <span class="math-container">${replicateId}.2.trimmed.fastq -S $</span>{replicateId}.Aligned.sam 2&gt; <span class="math-container">${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - &gt; $</span>{replicateId}.bam
  
  &quot;&quot;&quot;
  
  }
  
  
  
  workflow{
  
  reads_ch= Channel.fromFilePairs(params.reads)
  index_ch= Channel.fromPath(params.index)
  Channel.fromFilePairs(params.reads).view()
  index_ch.view()
  cut_adapt_trim(reads_ch)
  align_to_genome(index_ch,cut_adapt_trim.out)
  
  }
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <nextflow>
Title: How do I start a nextflow run from last executed command
ThreadText: # How do I start a nextflow run from last executed command
  
  (Asked by: username_0 on 2024-10-06)
  
  I am trying to use nextflow to process some ATAC seq data. The data is paired end and the files are huge. So, I first trimmed the adapters and now I want to align them to the genome.
  My script is shown below. Now, I want to align them using bowtie2. However, when I run the script, the trimming is being rerun again instead of starting at the alignment part.
  Below is my call to nextflow at the command line
  
  ```
  nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results -resume
  
  ```
  
  I was wondering what I am doing wrong such that the resume flag is not working?
  
  Thanks for any suggestions!
  
  ```
  #!/usr/bin/env nextflow
  
  nextflow.enable.dsl = 2
  
  params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
  params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
  
  /*
  * Trim atac seq reads to align to genome with cutadapt
  */
  
  process cut_adapt_trim {
  
  publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
  input:
  
  tuple val(replicateId), path(reads)
  
  output:
  
  tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
  tuple val(replicateId), path("${replicateId}.2.trimmed.fastq"), emit: trimmed_pair_2
  
  script:
  """
  
  cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o ${replicateId}.1.trimmed.fastq -p ${replicateId}.2.trimmed.fastq ${reads}
  
  """
  
  }
  
  /*
  * Align trimmed atac seq reads to genome
  */
  
  process align_to_genome {
  
  publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
  input:
  
  path index
  
  tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.1.trimmed.fastq")
  tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.2.trimmed.fastq")
  
  output:
  
  tuple val(replicateId), path("${replicateId}.sam"), emit:aligned_sam
  tuple val(replicateId), path("${replicateId}.bam"), emit:aligned_bam
  
  script:
  
  """
  def bwt_index = index[0].simpleName
  
  /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x ${bwt_index} -1 ${replicateId}.1.trimmed.fastq -2 ${replicateId}.2.trimmed.fastq -S ${replicateId}.Aligned.sam 2> ${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - > ${replicateId}.bam
  
  """
  
  }
  
  workflow{
  
  reads_ch= Channel.fromFilePairs(params.reads)
  index_ch= Channel.fromPath(params.index)
  Channel.fromFilePairs(params.reads).view()
  index_ch.view()
  cut_adapt_trim(reads_ch)
  align_to_genome(index_ch,cut_adapt_trim.out)
  
  }
  ```
  
  ```
  
  **Tags:** nextflow
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-10-06. Score: 2
  
  Your [task cache](https://www.nextflow.io/docs/stable/cache-and-resume.html#task-cache) is being invalidated because your [process inputs are non-deterministic](https://www.nextflow.io/docs/stable/cache-and-resume.html#non-deterministic-process-inputs). Like in the example, the solution is to [join](https://www.nextflow.io/docs/stable/operator.html#join) the two output channels before process invocation. Another way is to simply output the trimmed reads in a single channel, for example:
  
  ```
  process cut_adapt_trim {
  
      tag "$replicateId"
  
      publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
      conda 'cutadapt'
  
      input:
      tuple val(replicateId), path(reads)
      
      output:
      tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), path("${replicateId}.2.trimmed.fastq")
      
      script:
      """
      cutadapt \\
          -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG \\
          -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG \\
          -o "${replicateId}.1.trimmed.fastq" \\
          -p "${replicateId}.2.trimmed.fastq" \\
          ${reads}
      """
  }
  
  ```
  
  ```
  process align_to_genome {
  
      tag "$replicateId"
  
      publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
      conda 'bowtie2'
  
      input:
      tuple val(replicateId), path(fastq_1), path(fastq_2)
      path index
  
      output:
      tuple val(replicateId), path("${replicateId}.bam")
  
      script:
      def bwt_index = index[0].simpleName
  
      """
      bowtie2 \\
          -p 64 \\
          -q \\
          -x "${bwt_index}" \\
          -1 "${fastq_1}" \\
          -2 "${fastq_2}" |
      samtools view \\
          -b \\
          -o "${replicateId}.bam" \\
          -
      """
  }
  
  ```
  
  ```
  workflow{
  
      reads_ch = Channel.fromFilePairs( params.reads )
      index_ch = Channel.fromPath( params.index )
  
      cut_adapt_trim( reads_ch )
  
      align_to_genome( cut_adapt_trim.out, index_ch )
  
      align_to_genome.out.view()
  
  }
  
  ```
  
  Run using:
  
  ```
  nextflow run run_nextflow_ATACseq.nf -with-conda -resume 
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2024-10-06. Score: 2
  
  You can achieve your desired checkpointing with some if statements and a user defined flag pretty easily. Something like:
  
  ```
  #!/usr/bin/env nextflow
  nextflow.enable.dsl = 2
  
  params.skip_trim = null 
  params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
  params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
  params.trimmed_dir='01_trimmed'
  
  process cut_adapt_trim {
     publishDir ${params.trimmed_dir}, mode: 'copy'
  
     input:
     tuple val(replicateId), path(reads)
  
     output:
     tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
  tuple val(replicateId), path("${replicateId}.2.trimmed.fastq"), emit: trimmed_pair_2
  
     script:
     """
     cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o ${replicateId}.1.trimmed.fastq -p ${replicateId}.2.trimmed.fastq ${reads}
     """
  }
  
  process align_to_genome {
     publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'
  
     input:
     path index
     tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.1.trimmed.fastq")
  tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.2.trimmed.fastq")
  
     output:
     tuple val(replicateId), path("${replicateId}.sam"), emit:aligned_sam
  tuple val(replicateId), path("${replicateId}.bam"), emit:aligned_bam
  
     script:
     """
     def bwt_index = index[0].simpleName
  
     /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x ${bwt_index} -1 ${replicateId}.1.trimmed.fastq -2 ${replicateId}.2.trimmed.fastq -S ${replicateId}.Aligned.sam 2> ${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - > ${replicateId}.bam
  
     """
  }
  
  workflow{
     index_ch= Channel.fromPath(params.index)
     if( params.skip_trim == false ) {
         reads_ch= Channel.fromFilePairs(params.reads)
         cut_adapt_trim(reads_ch)
         align_to_genome(index_ch,cut_adapt_trim.out)
     } else if( params.skip_trim ) {
         trim_ch= Channel.fromFilePairs("${params.trimmed_dir}/*_{1,2}.trimmed.fastq")
         align_to_genome(index_ch,trim_ch)
     }
  }
  
  ```
  
  Then to execute the skip you would use:
  
  ```
  nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results --skip_trim
  
  ```
  
  Play around with the `trimmed_dir` variable, I don't know what you are mounting to. But using variables for a lot of the paths and options into the params section makes it easier to adjust in the future for other files.
  
  There is another point I would like to address. Nextflow emits and inputs files in random order. So by splitting the R1 and R2 reads into separate channels, you are going to have a lot of mixed samples as your output. Make the output a single tuple: `tuple val(id), path(r1), path(r2), emit: trimmed_reads`
  
  > **Comments:**
  > **username_0**: Thank you again for the feedback. Very helpful!  
  
  ---
  
  