Answers.Body: <p>Ok, I think the Slurm Executor is the preferred way, but there is an issue wrt my cluster (the last version does not work, but 3 version back does), I have filed an issue on GitHub: <a href="https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157" rel="nofollow noreferrer">https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157</a>, hope to resolve this soon, I'll put my efforts towards getting the Slurm executor working well for me.</p>
  <p>FYI, it turns out that in my case I probably shouldn't send jobs to other partitions in my particular case.</p>
  <p>Moreover, this code from above was a mistake, the Slurm executor does not accept this:</p>
  <pre class="lang-bash prettyprint-override"><code>-o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'
  </code></pre>
  <p>And probably for the better because once you get started with job-grouping that line does not make sense anymore (there are now multiple rules or samples per job!).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-27T21:18:48.313Z
Answers.LastActivityDate: 2024-10-28T07:53:00.330Z
Answers.LastEditDate: 2024-10-28T07:53:00.330Z
Answers.UserDisplayName: Freek
Body: <p>I've been experimenting with Snakemake on a (multi-partition) Slurm cluster (Snellius, the Dutch super computer).</p>
  <p>I have switched to Snakemake 8 so that I could make easier use of profiles. So I am using executors. I have tried 2 so far, snakemake-executor-plugin-cluster-generic and snakemake-executor-plugin-slurm.</p>
  <p>They both have down-sides, using snakemake-executor-plugin-cluster-generic I can't seem to be able to select the partition that I want, I tried this:</p>
  <pre class="lang-yaml prettyprint-override"><code>executor: cluster-generic
  cluster-generic-submit-cmd:
    sbatch
      --cpus-per-task=16
      --job-name={rule}-{jobid}
      --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
      --parsable
  restart-times: 3
  max-jobs-per-second: 10
  max-status-checks-per-second: 1
  local-cores: 1
  latency-wait: 60
  jobs: 100 # Check what the max is
  keep-going: True
  rerun-incomplete: True
  printshellcmds: True
  # scheduler: greedy
  use-conda: True
  
  set-threads:
    salmon: 16
  
  set-resources:
    fastqc:
      partition: staging
      # mem_mb: 100
  </code></pre>
  <p>In the submitted job I indeed see: <code>resources: tmpdir=&lt;TBD&gt;, partition=staging</code></p>
  <p>I also tried:</p>
  <pre class="lang-yaml prettyprint-override"><code>set-resources:
    fastqc:
      slurm_partition: staging
  </code></pre>
  <p>Again, I see it scrolling by under &quot;resources&quot; however, using squeue and the logs I see that the jobs are still executed on the standard partition (rome, which has minimum of 16 cores, I need just 1).</p>
  <p>So I switched to snakemake-executor-plugin-slurm using the following config.yaml:</p>
  <pre class="lang-yaml prettyprint-override"><code>executor: slurm
  default-resources:
    # not necessary if not sure about `slurm_account`, snakemake will try to infer this using `sacctmgr`
    slurm_partition: &quot;rome&quot;
    slurm_extra: &quot;' -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'&quot;
  printshellcmds: True
  jobs: 100
  restart-times: 3
  latency-wait: 60
  rerun-incomplete: True
  use-conda: True
  set-threads:
      salmon: 16
  
  set-resources:
      # rule specific resources
    fastqc:
      slurm_partition: staging
  </code></pre>
  <p>I can send jobs to other partitions! However, these jobs finish and the pipeline never moves past the initial steps as if snakemake never get the &quot;job finished&quot; signal. Also, the jobnames are just integers and the cluster_outputs does not show any logs. When I ctrl-c the main snakemake process I get this error:</p>
  <pre class="lang-bash prettyprint-override"><code>[Tue Oct 15 17:13:55 2024]
  rule fastqc:
      input: fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz
      output: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip
      jobid: 16
      benchmark: benchmark/0053_P2017BB3S19R_S1_R1.fastqc.fastqc.benchmark.tsv
      reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html
      wildcards: sample=0053_P2017BB3S19R_S1, read=R1
      resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=&lt;TBD&gt;, slurm_account=eccdcdc, slurm_partition=staging, slurm_extra= -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err
  
  
          fastqc fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz --outdir=qc/fastqc
          
  No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
  Job 16 has been submitted with SLURM jobid sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
  sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
  sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
  sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
  8177956 (log: /gpfs/home5/me/temp/test_pipeline/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R1/sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
  sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
  sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
  sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
  8177956.log).
  ^CTerminating processes on user request, this might take some time.
  WorkflowError:
  Unable to cancel jobs with scancel (exit code 127): scancel: unrecognized option '--exclusive'
  Try &quot;scancel --help&quot; for more information
  /bin/sh: line 2: sbatch:: command not found
  /bin/sh: line 3: sbatch:: command not found
  /bin/sh: line 4: sbatch:: command not found
  /bin/sh: line 9: 8177952: command not found
  /bin/sh: line 10: sbatch:: command not found
  /bin/sh: line 11: sbatch:: command not found
  /bin/sh: line 12: sbatch:: command not found
  /bin/sh: line 17: 8177954: command not found
  /bin/sh: line 18: sbatch:: command not found
  /bin/sh: line 19: sbatch:: command not found
  /bin/sh: line 20: sbatch:: command not found
  /bin/sh: line 25: 8177956: command not found
  (snaqs) [me@int4 test_pipeline]$ 
  </code></pre>
  <p>How can I get best of both worlds? The generic cluster executor works perfectly except for sending job to another the partition, the slurm executor won't finish, most likely I'm missing something.</p>
  <p>Thank you!</p>
  <p>EDIT:</p>
  <p>With the cluster-generic executor I can change the partition succesfully by modifying the sbatch command like this:</p>
  <pre class="lang-yaml prettyprint-override"><code>executor: cluster-generic
  cluster-generic-submit-cmd:
    sbatch
      --cpus-per-task=16
      --job-name={rule}-{jobid}
      --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
      --parsable
      --partition=staging
  </code></pre>
  <p>Now the question is, how to specify it per rule...</p>
  <p>EDIT:</p>
  <p>It seems that when submitting with the slurm executor all jobs have the same name, that could be an issue?</p>
  <pre class="lang-bash prettyprint-override"><code>$ squeue -o&quot;%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C %k&quot;
    JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS COMMENT
  8178848      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S20R_S2
  8178845      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S19R_S1
  8178846   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R2
  8178847   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R1
  8178849   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R2
  8178850   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R1
  </code></pre>
  <p>Oh no of course not, the JOBID is different. I'll leave this here none-the-less.</p>
  
Tags: <snakemake><slurm>
Title: Snakemake on a Slurm cluster
ThreadText: # Snakemake on a Slurm cluster
  
  (Asked by: username_0 on 2024-10-15)
  
  I've been experimenting with Snakemake on a (multi-partition) Slurm cluster (Snellius, the Dutch super computer).
  
  I have switched to Snakemake 8 so that I could make easier use of profiles. So I am using executors. I have tried 2 so far, snakemake-executor-plugin-cluster-generic and snakemake-executor-plugin-slurm.
  
  They both have down-sides, using snakemake-executor-plugin-cluster-generic I can't seem to be able to select the partition that I want, I tried this:
  
  ```yaml
  executor: cluster-generic
  cluster-generic-submit-cmd:
    sbatch
      --cpus-per-task=16
      --job-name={rule}-{jobid}
      --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
      --parsable
  restart-times: 3
  max-jobs-per-second: 10
  max-status-checks-per-second: 1
  local-cores: 1
  latency-wait: 60
  jobs: 100 # Check what the max is
  keep-going: True
  rerun-incomplete: True
  printshellcmds: True
  # scheduler: greedy
  use-conda: True
  
  set-threads:
    salmon: 16
  
  set-resources:
    fastqc:
      partition: staging
      # mem_mb: 100
  
  ```
  
  In the submitted job I indeed see: `resources: tmpdir=<TBD>, partition=staging`
  
  I also tried:
  
  ```yaml
  set-resources:
    fastqc:
      slurm_partition: staging
  
  ```
  
  Again, I see it scrolling by under "resources" however, using squeue and the logs I see that the jobs are still executed on the standard partition (rome, which has minimum of 16 cores, I need just 1).
  
  So I switched to snakemake-executor-plugin-slurm using the following config.yaml:
  
  ```yaml
  executor: slurm
  default-resources:
    # not necessary if not sure about `slurm_account`, snakemake will try to infer this using `sacctmgr`
    slurm_partition: "rome"
    slurm_extra: "' -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'"
  printshellcmds: True
  jobs: 100
  restart-times: 3
  latency-wait: 60
  rerun-incomplete: True
  use-conda: True
  set-threads:
      salmon: 16
  
  set-resources:
      # rule specific resources
    fastqc:
      slurm_partition: staging
  
  ```
  
  I can send jobs to other partitions! However, these jobs finish and the pipeline never moves past the initial steps as if snakemake never get the "job finished" signal. Also, the jobnames are just integers and the cluster\_outputs does not show any logs. When I ctrl-c the main snakemake process I get this error:
  
  ```bash
  [Tue Oct 15 17:13:55 2024]
  rule fastqc:
      input: fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz
      output: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip
      jobid: 16
      benchmark: benchmark/0053_P2017BB3S19R_S1_R1.fastqc.fastqc.benchmark.tsv
      reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html
      wildcards: sample=0053_P2017BB3S19R_S1, read=R1
      resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=eccdcdc, slurm_partition=staging, slurm_extra= -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err
  
          fastqc fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz --outdir=qc/fastqc
          
  No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
  Job 16 has been submitted with SLURM jobid sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
  sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
  sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
  sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
  8177956 (log: /gpfs/home5/me/temp/test_pipeline/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R1/sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
  sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
  sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
  sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
  8177956.log).
  ^CTerminating processes on user request, this might take some time.
  WorkflowError:
  Unable to cancel jobs with scancel (exit code 127): scancel: unrecognized option '--exclusive'
  Try "scancel --help" for more information
  /bin/sh: line 2: sbatch:: command not found
  /bin/sh: line 3: sbatch:: command not found
  /bin/sh: line 4: sbatch:: command not found
  /bin/sh: line 9: 8177952: command not found
  /bin/sh: line 10: sbatch:: command not found
  /bin/sh: line 11: sbatch:: command not found
  /bin/sh: line 12: sbatch:: command not found
  /bin/sh: line 17: 8177954: command not found
  /bin/sh: line 18: sbatch:: command not found
  /bin/sh: line 19: sbatch:: command not found
  /bin/sh: line 20: sbatch:: command not found
  /bin/sh: line 25: 8177956: command not found
  (snaqs) [me@int4 test_pipeline]$ 
  
  ```
  
  How can I get best of both worlds? The generic cluster executor works perfectly except for sending job to another the partition, the slurm executor won't finish, most likely I'm missing something.
  
  Thank you!
  
  EDIT:
  
  With the cluster-generic executor I can change the partition succesfully by modifying the sbatch command like this:
  
  ```yaml
  executor: cluster-generic
  cluster-generic-submit-cmd:
    sbatch
      --cpus-per-task=16
      --job-name={rule}-{jobid}
      --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
      --parsable
      --partition=staging
  
  ```
  
  Now the question is, how to specify it per rule...
  
  EDIT:
  
  It seems that when submitting with the slurm executor all jobs have the same name, that could be an issue?
  
  ```bash
  $ squeue -o"%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C %k"
    JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS COMMENT
  8178848      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S20R_S2
  8178845      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S19R_S1
  8178846   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R2
  8178847   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R1
  8178849   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R2
  8178850   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R1
  
  ```
  
  Oh no of course not, the JOBID is different. I'll leave this here none-the-less.
  
  **Tags:** snakemake, slurm
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-10-27. Score: 0 | Accepted answer
  
  Ok, I think the Slurm Executor is the preferred way, but there is an issue wrt my cluster (the last version does not work, but 3 version back does), I have filed an issue on GitHub: <https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157>, hope to resolve this soon, I'll put my efforts towards getting the Slurm executor working well for me.
  
  FYI, it turns out that in my case I probably shouldn't send jobs to other partitions in my particular case.
  
  Moreover, this code from above was a mistake, the Slurm executor does not accept this:
  
  ```bash
  -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'
  
  ```
  
  And probably for the better because once you get started with job-grouping that line does not make sense anymore (there are now multiple rules or samples per job!).
  
  
  ---
  
  