Answers.Body: <p><strong>Simple answer</strong> The answer is this should be done via due process. A 1 hour time limit is harsh on any system. I'd talk with your line manager to justify your CPU time and then put the case to the Slurm administrator. This is really about management structure than tech. Really 1 hour versus 1.5 hours or less should not be an issue and your individual account can be set to accommodate that easily.</p>
  <p>Please do avoid hacking your system.</p>
  <p>Just for background snakemake is cool(ish) because it uses conda, but thats also an issue in HPC (conda programs are not HPC optimised). NextFlow uses Docker containers.</p>
  <p><strong>Background</strong> You appear want snakemake set up independent of the Slurm account manager (the software that controls queue access) and therefore override it. Sure there's an independent setup for snakemake, but that shouldn't be a workaround permitted, i.e. to override the Slurm account manager for your account. This job is definitely within your account/group restrictions because it has been stopped exactly on the hour (thats Slurm at work). Cluster usage will be tracked BTW.</p>
  <p><strong>Containers within containers</strong>
  The thing that is unusual here is the code used because snakemake and NextFlow do the same thing, i.e. containerise the job, snakemake via Conda. The thing here - just my brief reading of your code - is that a snakemake container is being called and within that container a Nextflow container is being deployed. Thus you've a container within a container. Thats unnecessary. I'd run either snakemake or NextFlow, rather than NextFlow within snakemake.</p>
  <p>The reason a container is run is to protect the system from your code, thus you only need one container not a nested container.</p>
  <hr />
  <p>If the Slurm administrator will not increase your CPU time above 1 hour per job (I believe they will if your give them good justification) then breaking up GRCh38 into two or three smaller files and performing the calculation against each is one solution to the 1 hour limit. If thats the way you want to go and not sure how to do it via code, just ask another question.</p>
  <p><strong>Note</strong> Two things here Slurm has an &quot;account manager&quot; thats a command that accesses each user or group to set limits, grant privilege etc ... to the Slurm cluster. There is also the Slurm administrator. Thats the person in charge. Just to be clear &quot;Slurm account manager&quot; is the name of the software and not a person.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-01T17:40:34.687Z
Answers.LastActivityDate: 2024-05-01T21:14:36.510Z
Answers.LastEditDate: 2024-05-01T21:14:36.510Z
Answers.UserDisplayName: M__
Answers.Body: <p>I think the actual issue here is just with the command line:</p>
  <pre><code>snakemake --slurm --jobs 23 --use-conda --conda-frontend conda --cluster &quot;sbatch --time=24:00:00&quot; nextflow/score/aggregated_scores.txt.gz -n
  </code></pre>
  <p>It includes both <code>--slurm</code> and <code>--cluster</code> options and these are actually mutually exclusive options. The help text for <code>--slurm</code> says (for snakemake 7):</p>
  <pre><code>Execute snakemake rules as SLURM batch jobs according
  to their 'resources' definition. SLURM resources as
  'partition', 'ntasks', 'cpus', etc. need to be defined
  per rule within the 'resources' definition.
  </code></pre>
  <p>So Snakemake is just ignoring anything you put in <code>--cluster</code> because you have the <code>--slurm</code> option.</p>
  <p>So you need to either specify the timeout within the job resources in the Snakefile, or else rely on the <code>--cluster</code> option to run your jobs with the right options. Neither of these is as easy/elegant as you might hope. I've always run my SLURM jobs with the <code>--drmaa</code> method and they generally behave themselves, once you have the right DRMAA driver.</p>
  <p>You may also be able to just set the environment variable &quot;SLURM_TIMELIMIT&quot; which avoids fiddling with Snakemake configuration. This does not affect DRMAA jobs but should apply where Snakemake is just calling sbatch/srun to start the jobs.</p>
  <p>Also, congrats on using Snakemake and NextFlow and Conda and Singularity in one single workflow :-)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-19T16:30:39.573Z
Answers.LastActivityDate: 2024-06-19T16:30:39.573Z
Answers.UserDisplayName: Tim Booth
Body: <p>I am running PGS Calculator on a High-Performance Computer with the operating system GNU/Linux.</p>
  <p>I have an executable script named pgs_hack.slurm containing the following code:</p>
  <pre><code>#!/bin/bash
  #SBATCH --job-name=pgs-calc
  #SBATCH --partition=nodes
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=8
  #SBATCH --mem=64G
  #SBATCH --time=24:00:00
  #SBATCH --output=pgs-calc_%j.log
  #SBATCH --error=pgs-calc_%j.err
  
  # Load necessary modules
  module load nextflow
  
  # Run Nextflow
  nextflow run pgscatalog/pgsc_calc 
          -profile singularity \
          --input samplesheet.csv \
          --target_build GRCh38 \
          --min_overlap 0.01 \
          --trait_efo EFO_0000537 \
          --outdir EFO_0000537/
  </code></pre>
  <p>This script finished successfully in about 90 minutes.</p>
  <p>I then wrapped the PGS Calculator in a snakemake rule:</p>
  <pre><code>rule pgscalc_from_pgs_catalog_efo:
      input:
          samplesheet = 'samplesheet.csv',
      output:
          aggregated_scores = 'nextflow/score/aggregated_scores.txt.gz',
      wildcard_constraints:
          EFO_ID = 'EFO_\\d{7}',
      log:
          'pgscalc_from_pgs_catalog_efo.log',
      conda:
          'envs/singularity=3.8.6.yaml'
      threads: 8
      resources:
          mem_mb=64000,
      shell:
          'nextflow run pgscatalog/pgsc_calc -profile singularity --input {input.samplesheet} --target_build GRCh38 --min_overlap 0.50 --trait_efo {wildcards.EFO_ID} --outdir nextflow/ &gt; {log} 2&gt;&amp;1'
  </code></pre>
  <p>I then launched the pipeline using the following snakemake command:</p>
  <pre><code>snakemake --slurm --jobs 23 --use-conda --conda-frontend conda --cluster &quot;sbatch --time=24:00:00&quot; nextflow/score/aggregated_scores.txt.gz -n
  </code></pre>
  <p>but I got the following error:</p>
  <pre><code>...
  [Tue Apr 30 16:49:38 2024]
  ...
  slurmstepd: error: *** STEP 38981.0 ON node01 CANCELLED AT 2024-04-30T17:49:45 DUE TO TIME LIMIT ***
  srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
  slurmstepd: error: *** JOB 38981 ON node01 CANCELLED AT 2024-04-30T17:49:45 DUE TO TIME LIMIT ***
  Will exit after finishing currently running jobs (scheduler).
  Will exit after finishing currently running jobs (scheduler).
  </code></pre>
  <p>The time limit set by our system administrator is 1 hour. But why isn't my snakemake sbatch option not overriding this default?</p>
  <p>Thanks, and best wishes,</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-02T19:06:25.127Z
Comments.Text: Do you mean the default time limit is 1 hour unless you specify a different time in sbatch options, or that 1 hour is the absolute maximum time? If it's the latter, I don't think you can get around it.
Comments.UserDisplayName: Cloudberry
Tags: <snakemake><hpc><slurm>
Title: How can I override the time limit set by my administrator on SLURM using snakemake?
ThreadText: # How can I override the time limit set by my administrator on SLURM using snakemake?
  
  (Asked by: username_0 on 2024-05-01)
  
  I am running PGS Calculator on a High-Performance Computer with the operating system GNU/Linux.
  
  I have an executable script named pgs\_hack.slurm containing the following code:
  
  ```
  #!/bin/bash
  #SBATCH --job-name=pgs-calc
  #SBATCH --partition=nodes
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=8
  #SBATCH --mem=64G
  #SBATCH --time=24:00:00
  #SBATCH --output=pgs-calc_%j.log
  #SBATCH --error=pgs-calc_%j.err
  
  # Load necessary modules
  module load nextflow
  
  # Run Nextflow
  nextflow run pgscatalog/pgsc_calc 
          -profile singularity \
          --input samplesheet.csv \
          --target_build GRCh38 \
          --min_overlap 0.01 \
          --trait_efo EFO_0000537 \
          --outdir EFO_0000537/
  
  ```
  
  This script finished successfully in about 90 minutes.
  
  I then wrapped the PGS Calculator in a snakemake rule:
  
  ```
  rule pgscalc_from_pgs_catalog_efo:
      input:
          samplesheet = 'samplesheet.csv',
      output:
          aggregated_scores = 'nextflow/score/aggregated_scores.txt.gz',
      wildcard_constraints:
          EFO_ID = 'EFO_\\d{7}',
      log:
          'pgscalc_from_pgs_catalog_efo.log',
      conda:
          'envs/singularity=3.8.6.yaml'
      threads: 8
      resources:
          mem_mb=64000,
      shell:
          'nextflow run pgscatalog/pgsc_calc -profile singularity --input {input.samplesheet} --target_build GRCh38 --min_overlap 0.50 --trait_efo {wildcards.EFO_ID} --outdir nextflow/ > {log} 2>&1'
  
  ```
  
  I then launched the pipeline using the following snakemake command:
  
  ```
  snakemake --slurm --jobs 23 --use-conda --conda-frontend conda --cluster "sbatch --time=24:00:00" nextflow/score/aggregated_scores.txt.gz -n
  
  ```
  
  but I got the following error:
  
  ```
  ...
  [Tue Apr 30 16:49:38 2024]
  ...
  slurmstepd: error: *** STEP 38981.0 ON node01 CANCELLED AT 2024-04-30T17:49:45 DUE TO TIME LIMIT ***
  srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
  slurmstepd: error: *** JOB 38981 ON node01 CANCELLED AT 2024-04-30T17:49:45 DUE TO TIME LIMIT ***
  Will exit after finishing currently running jobs (scheduler).
  Will exit after finishing currently running jobs (scheduler).
  
  ```
  
  The time limit set by our system administrator is 1 hour. But why isn't my snakemake sbatch option not overriding this default?
  
  Thanks, and best wishes,
  
  **Tags:** snakemake, hpc, slurm
  
  > **Comments:**
  > **username_1**: Do you mean the default time limit is 1 hour unless you specify a different time in sbatch options, or that 1 hour is the absolute maximum time? If it's the latter, I don't think you can get around it.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-05-01. Score: 2
  
  **Simple answer** The answer is this should be done via due process. A 1 hour time limit is harsh on any system. I'd talk with your line manager to justify your CPU time and then put the case to the Slurm administrator. This is really about management structure than tech. Really 1 hour versus 1.5 hours or less should not be an issue and your individual account can be set to accommodate that easily.
  
  Please do avoid hacking your system.
  
  Just for background snakemake is cool(ish) because it uses conda, but thats also an issue in HPC (conda programs are not HPC optimised). NextFlow uses Docker containers.
  
  **Background** You appear want snakemake set up independent of the Slurm account manager (the software that controls queue access) and therefore override it. Sure there's an independent setup for snakemake, but that shouldn't be a workaround permitted, i.e. to override the Slurm account manager for your account. This job is definitely within your account/group restrictions because it has been stopped exactly on the hour (thats Slurm at work). Cluster usage will be tracked BTW.
  
  **Containers within containers**
  The thing that is unusual here is the code used because snakemake and NextFlow do the same thing, i.e. containerise the job, snakemake via Conda. The thing here - just my brief reading of your code - is that a snakemake container is being called and within that container a Nextflow container is being deployed. Thus you've a container within a container. Thats unnecessary. I'd run either snakemake or NextFlow, rather than NextFlow within snakemake.
  
  The reason a container is run is to protect the system from your code, thus you only need one container not a nested container.
  
  ---
  
  If the Slurm administrator will not increase your CPU time above 1 hour per job (I believe they will if your give them good justification) then breaking up GRCh38 into two or three smaller files and performing the calculation against each is one solution to the 1 hour limit. If thats the way you want to go and not sure how to do it via code, just ask another question.
  
  **Note** Two things here Slurm has an "account manager" thats a command that accesses each user or group to set limits, grant privilege etc ... to the Slurm cluster. There is also the Slurm administrator. Thats the person in charge. Just to be clear "Slurm account manager" is the name of the software and not a person.
  
  
  ---
  
  ### Answer by username_3 on 2024-06-19. Score: 1
  
  I think the actual issue here is just with the command line:
  
  ```
  snakemake --slurm --jobs 23 --use-conda --conda-frontend conda --cluster "sbatch --time=24:00:00" nextflow/score/aggregated_scores.txt.gz -n
  
  ```
  
  It includes both `--slurm` and `--cluster` options and these are actually mutually exclusive options. The help text for `--slurm` says (for snakemake 7):
  
  ```
  Execute snakemake rules as SLURM batch jobs according
  to their 'resources' definition. SLURM resources as
  'partition', 'ntasks', 'cpus', etc. need to be defined
  per rule within the 'resources' definition.
  
  ```
  
  So Snakemake is just ignoring anything you put in `--cluster` because you have the `--slurm` option.
  
  So you need to either specify the timeout within the job resources in the Snakefile, or else rely on the `--cluster` option to run your jobs with the right options. Neither of these is as easy/elegant as you might hope. I've always run my SLURM jobs with the `--drmaa` method and they generally behave themselves, once you have the right DRMAA driver.
  
  You may also be able to just set the environment variable "SLURM\_TIMELIMIT" which avoids fiddling with Snakemake configuration. This does not affect DRMAA jobs but should apply where Snakemake is just calling sbatch/srun to start the jobs.
  
  Also, congrats on using Snakemake and NextFlow and Conda and Singularity in one single workflow :-)
  
  
  ---
  
  