Answers.Body: <p>Unfortunately, this remains an open issue with Nextflow: <a href="https://github.com/nextflow-io/nextflow/issues/452" rel="nofollow noreferrer">https://github.com/nextflow-io/nextflow/issues/452</a></p>
  <p>However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.</p>
  <p>Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but <a href="https://github.com/ridgelab/Kmer-SSR" rel="nofollow noreferrer">Kmer-SSR</a> <em>can</em> indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:</p>
  <pre><code>process kmer_ssr {
  
      cpus 4
  
      input:
      tuple val(sample), path(fastq)
  
      output:
      tuple val(sample), path(&quot;${sample}.tsv.gz&quot;)
  
      &quot;&quot;&quot;
      zcat -f &quot;<span class="math-container">${fastq}" \\
          | sed -n '1~4s/^@/&gt;/p; 2~4p' \\
          | kmer-ssr -t $</span>{task.cpus} \\
          | gzip \\
          &gt; &quot;${sample}.tsv.gz&quot;
      &quot;&quot;&quot;
  }
  </code></pre>
  <p>Since the above uses a pipeline, make sure to add the <code>pipefail</code> to your command. One way is to simply apply it to all processes by adding the following to your <code>nextflow.config</code>:</p>
  <pre><code>process {
  
    shell = [ '/bin/bash', '-euo', 'pipefail' ]
  }
  </code></pre>
  <p>From the <code>bash</code> man page:</p>
  <blockquote>
  <p>pipefail</p>
  <p>If set, the return value of a pipeline is the value of the last
  (rightmost) command to exit with a non-zero status, or zero if all
  commands in the pipeline exit successfully. This option is disabled by
  default.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-12T03:25:57.977Z
Answers.LastActivityDate: 2023-02-12T03:25:57.977Z
Answers.UserDisplayName: Steve
Body: <p>In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.</p>
  <p>In my pipeline I use kmer-ssr to count kmers present on my genome sample.</p>
  <p>First I create a reads_ch = Channel.fromSRA</p>
  <p>In this channel there are 12 files (. fastq.gz) of 20 Gb each one.</p>
  <p>Then the first process unzips them and transforms them in files.fasta.</p>
  <p>Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.</p>
  <p>The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.</p>
  <p>The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-11T13:26:17.107Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-11T13:31:28.373Z
Comments.Text: Certainly this will be doable. Could you provide the code? Or else sufficient amounts of code to describe the nested loop? I will provide a link below as typical example of a `NextFlow` question.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-11T13:33:01.407Z
Comments.Text: So this is yesterdays question [here](https://bioinformatics.stackexchange.com/questions/20536/nextflow-tuple-from-csv-cols/20537#20537) as an example of a standard `NextFlow` question on the site.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-13T00:59:55.910Z
Comments.Text: Look into the "temporary" attribute of the inputs that should help you in the right direction.
Comments.UserDisplayName: raygozag
Tags: <fasta><fastq><nextflow><k-mer><workflow-management>
Title: Remove files after process is terminated
ThreadText: # Remove files after process is terminated
  
  (Asked by: username_0 on 2023-02-11)
  
  In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.
  
  In my pipeline I use kmer-ssr to count kmers present on my genome sample.
  
  First I create a reads\_ch = Channel.fromSRA
  
  In this channel there are 12 files (. fastq.gz) of 20 Gb each one.
  
  Then the first process unzips them and transforms them in files.fasta.
  
  Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.
  
  The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.
  
  The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process
  
  **Tags:** fasta, fastq, nextflow, k-mer, workflow-management
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: Certainly this will be doable. Could you provide the code? Or else sufficient amounts of code to describe the nested loop? I will provide a link below as typical example of a `NextFlow` question.  
  > **username_2**: So this is yesterdays question [here](https://bioinformatics.stackexchange.com/questions/20536/nextflow-tuple-from-csv-cols/20537#20537) as an example of a standard `NextFlow` question on the site.  
  > **username_3**: Look into the "temporary" attribute of the inputs that should help you in the right direction.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-02-12. Score: 0
  
  Unfortunately, this remains an open issue with Nextflow: <https://github.com/nextflow-io/nextflow/issues/452>
  
  However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.
  
  Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but [Kmer-SSR](https://github.com/ridgelab/Kmer-SSR) *can* indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:
  
  ```
  process kmer_ssr {
  
      cpus 4
  
      input:
      tuple val(sample), path(fastq)
  
      output:
      tuple val(sample), path("${sample}.tsv.gz")
  
      """
      zcat -f "${fastq}" \\
          | sed -n '1~4s/^@/>/p; 2~4p' \\
          | kmer-ssr -t ${task.cpus} \\
          | gzip \\
          > "${sample}.tsv.gz"
      """
  }
  
  ```
  
  Since the above uses a pipeline, make sure to add the `pipefail` to your command. One way is to simply apply it to all processes by adding the following to your `nextflow.config`:
  
  ```
  process {
  
    shell = [ '/bin/bash', '-euo', 'pipefail' ]
  }
  
  ```
  
  From the `bash` man page:
  
  > pipefail
  >
  > If set, the return value of a pipeline is the value of the last
  > (rightmost) command to exit with a non-zero status, or zero if all
  > commands in the pipeline exit successfully. This option is disabled by
  > default.
  
  
  ---
  
  