Answers.Body: <p>One Snakemake feature you can use is <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#protected-and-temporary-files" rel="nofollow noreferrer">temporary outputs</a>.</p>
  <p>If you make your outputs from the <code>copy_bam</code> rule temporary, they'll be cleared up after no other rule requires that file as an input.
  Your Snakefile would look something like this:</p>
  <pre class="lang-py prettyprint-override"><code>rule copy_bam:
      input:
          bam = os.path.join(config['bam_folder'], &quot;{sample}.gatk.sorted.MD.bam&quot;),
          idx = os.path.join(config['bam_folder'], &quot;{sample}.gatk.sorted.MD.bam.bai&quot;),
      output:
          bam = temp(&quot;{staging_area}/{sample}.bam&quot;),
          idx = temp(&quot;{staging_area}/{sample}.bam.bai&quot;),
      wildcard_constraints:
          sample=&quot;[A-Za-z0-9_]+&quot;
      resources:
          bam_copy=1
      shell:
          &quot;&quot;&quot;
          cp {input.bam} {output.bam}
          cp {input.idx} {output.idx}
          &quot;&quot;&quot;
  # omitted for brevity
  </code></pre>
  <p>You can then specify how many jobs run concurrently with the <code>--cores</code>/<code>--jobs</code> options to keep the space usage small.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T12:57:39.303Z
Answers.LastActivityDate: 2023-05-25T15:14:19.433Z
Answers.LastEditDate: 2023-05-25T15:14:19.433Z
Answers.UserDisplayName: James Hawley
Body: <p>I am trying to generate several snakemake rules. However, I would like to include some restrictions due to storage limitations</p>
  <p>I first would like to copy the the bamfiles, 1 for each sample, to a staging area. However, while I have over 50 samples, I can only process three samples at a time.</p>
  <p>Then I want to compare the bam file to chunked candidate files, 1 for each chromosome. This however, I would like to run on 23 cores per sample so that once each bam file is copied 69 cores are working.</p>
  <p>finally, if all chunks for a given sample have been processed I want to clean up the staging area, copy the bamfile for the next sample and process the chunks for that sample.</p>
  <p>My current implementation, see below, just continues copying bamfiles and crashes when I run out of strorage;</p>
  <pre><code>rule copy_bam:
      input:
          bamfile=os.path.join(config['bam_folder'], &quot;{sample}.gatk.sorted.MD.bam&quot;),
          bamindex=os.path.join(config['bam_folder'], &quot;{sample}.gatk.sorted.MD.bam.bai&quot;)
      output:
          staging_bamfile=&quot;{staging_area}/{sample}.bam&quot;,
          staging_bamindex=&quot;{staging_area}/{sample}.bam.bai&quot;
      wildcard_constraints:
          sample=&quot;[A-Za-z0-9_]+&quot;
      resources:
          bam_copy=1
      shell:
          &quot;&quot;&quot;
          cp {input.bamfile} {output.staging_bamfile}
          cp {input.bamindex} {output.staging_bamindex}
          &quot;&quot;&quot;
  
  rule process_chunks:
      input:
          chunkfile=&quot;{staging_area}/chunks/{chunk}_with_header.bcf.gz&quot;,
          bamfile=rules.copy_bam.output.staging_bamfile
      output:
          dynamic(&quot;{staging_area}/calls/{sample}/{chunk}.bcf&quot;)
  
      threads: 23
      shell:
          &quot;&quot;&quot;
          process chunks
          &quot;&quot;&quot;
  
  rule cleanup:
      input:
          bamfile=&quot;{staging_area}/{sample}.bam&quot;,
          bamindex=&quot;{staging_area}/{sample}.bam.bai&quot;,
          processed=lambda wildcards: expand(&quot;{staging_area}/calls/{sample}/{chunk}.bcf&quot;, chunk=get_chunks()[wildcards.sample], staging_area=config['staging_area'], sample=wildcards.sample)
      output:
          touch(&quot;{staging_area}/{sample}/cleanup_done.txt&quot;)
      shell:
          &quot;&quot;&quot;
          rm {input.bamfile}
          rm {input.bamindex}
          rm {input.processed}
          &quot;&quot;&quot;
  </code></pre>
  <p>run with: <code>snakemake --resources bam_copy=3 --cores 70</code></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-25T16:16:34.830Z
Comments.Text: One thing to note about any approach here that uses Snakemake's resources is that they're defined *per running job*, versus with the "in use until something happens in a later job" idea here. Jame's Hawley's suggestion of `temp()` should help, just note that I don't think the resource tracking even matters in that case (the resource is only "in use" while `copy_bam` is actively running.)
Comments.UserDisplayName: Jesse
Tags: <snakemake>
Title: snakemake restricting resources for specific rule
ThreadText: # snakemake restricting resources for specific rule
  
  (Asked by: username_0 on 2023-05-24)
  
  I am trying to generate several snakemake rules. However, I would like to include some restrictions due to storage limitations
  
  I first would like to copy the the bamfiles, 1 for each sample, to a staging area. However, while I have over 50 samples, I can only process three samples at a time.
  
  Then I want to compare the bam file to chunked candidate files, 1 for each chromosome. This however, I would like to run on 23 cores per sample so that once each bam file is copied 69 cores are working.
  
  finally, if all chunks for a given sample have been processed I want to clean up the staging area, copy the bamfile for the next sample and process the chunks for that sample.
  
  My current implementation, see below, just continues copying bamfiles and crashes when I run out of strorage;
  
  ```
  rule copy_bam:
      input:
          bamfile=os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam"),
          bamindex=os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam.bai")
      output:
          staging_bamfile="{staging_area}/{sample}.bam",
          staging_bamindex="{staging_area}/{sample}.bam.bai"
      wildcard_constraints:
          sample="[A-Za-z0-9_]+"
      resources:
          bam_copy=1
      shell:
          """
          cp {input.bamfile} {output.staging_bamfile}
          cp {input.bamindex} {output.staging_bamindex}
          """
  
  rule process_chunks:
      input:
          chunkfile="{staging_area}/chunks/{chunk}_with_header.bcf.gz",
          bamfile=rules.copy_bam.output.staging_bamfile
      output:
          dynamic("{staging_area}/calls/{sample}/{chunk}.bcf")
  
      threads: 23
      shell:
          """
          process chunks
          """
  
  rule cleanup:
      input:
          bamfile="{staging_area}/{sample}.bam",
          bamindex="{staging_area}/{sample}.bam.bai",
          processed=lambda wildcards: expand("{staging_area}/calls/{sample}/{chunk}.bcf", chunk=get_chunks()[wildcards.sample], staging_area=config['staging_area'], sample=wildcards.sample)
      output:
          touch("{staging_area}/{sample}/cleanup_done.txt")
      shell:
          """
          rm {input.bamfile}
          rm {input.bamindex}
          rm {input.processed}
          """
  
  ```
  
  run with: `snakemake --resources bam_copy=3 --cores 70`
  
  **Tags:** snakemake
  
  > **Comments:**
  > **username_1**: One thing to note about any approach here that uses Snakemake's resources is that they're defined *per running job*, versus with the "in use until something happens in a later job" idea here. Jame's username_2's suggestion of `temp()` should help, just note that I don't think the resource tracking even matters in that case (the resource is only "in use" while `copy_bam` is actively running.)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-25. Score: 2 | Accepted answer
  
  One Snakemake feature you can use is [temporary outputs](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#protected-and-temporary-files).
  
  If you make your outputs from the `copy_bam` rule temporary, they'll be cleared up after no other rule requires that file as an input.
  Your Snakefile would look something like this:
  
  ```py
  rule copy_bam:
      input:
          bam = os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam"),
          idx = os.path.join(config['bam_folder'], "{sample}.gatk.sorted.MD.bam.bai"),
      output:
          bam = temp("{staging_area}/{sample}.bam"),
          idx = temp("{staging_area}/{sample}.bam.bai"),
      wildcard_constraints:
          sample="[A-Za-z0-9_]+"
      resources:
          bam_copy=1
      shell:
          """
          cp {input.bam} {output.bam}
          cp {input.idx} {output.idx}
          """
  # omitted for brevity
  
  ```
  
  You can then specify how many jobs run concurrently with the `--cores`/`--jobs` options to keep the space usage small.
  
  
  ---
  
  