Answers.Body: <p>Got here from your slurm-excecutor slurm issue :)</p>
  <p>First thing's first, my use of snakemake is not for bioinformatics, I use it in a completely different field where we dont need that many small parallel jobs. I do use job grouping and to my understanding it should work as you intend it to.</p>
  <p>Also a disclaimer, I use snakemake 7.X. I still have not upgraded my workflow to work with 8.X, hence, I use the &quot;old&quot; slurm executor (the cookiecutter profile), but I hope my experience translates to 8.X if that's what you are working with.</p>
  <p>Now to your questions:
  The log file for a grouped rule should group all the small jobs submitted with the group. So in your case check:</p>
  <blockquote>
  <p>log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log</p>
  </blockquote>
  <p>If all the grouped rules are show in the log, then success. And yes, each group should only spawn one job submission with a unique jobID.
  It is kind of strange that the folder where the log is saved says &quot;rule&quot; in it, in my case logs are saved in a folder with the name of the group. This might be a difference between the profile and the executor.</p>
  <p>Something else that might be useful. After submitting the job, just spam squeue and see how your job queue looks like. Try with and without grouping, the difference should be quite clear.</p>
  <p>Creating groups with a wildcard in it is something I never tried, so can't help you with that one.</p>
  <p>Resource wise, remember you are being charged 16 cores anyways, which means you also get the memory allocation for that (By default 1.75gb per thread in a thin node). So if you have 4 rules that run in parallel and each of them needs... say 6gb, you can just specify the rule needs 4 threads and by default it will get 1.75*4gb of RAM, and since the 4 rules &quot;fit&quot; in the allocated node, they should run in parallel.
  In other words, I stopped worrying about optimizing for memory, and I now &quot;optimize&quot; by threads (even though I know many of my rules use just 1 core), and fine-tune from there!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T19:43:05.900Z
Answers.Text: Ok I got it to work, but only by defining goups in the rules, it doesn't work from the config yaml. I think I got the concept wrong, it seems that it only groups subsequent jobs? Can it also be used to make for example 16 small jobs be run each in 1 thread on a 16 core CPU?
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T20:41:10.503Z
Answers.Text: Ok, I think I have to keep in mind that you can only group tasks that are connected in the DAG, right?
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T21:23:13.913Z
Answers.Text: Ok,I'm running into this issue, will update here if I get it to work: https://github.com/snakemake/snakemake/issues/3165
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-30T16:13:25.533Z
Answers.LastActivityDate: 2024-10-30T16:19:53.550Z
Answers.LastEditDate: 2024-10-30T16:19:53.550Z
Answers.UserDisplayName: I Farias
Answers.Body: <p>According to this answer to an issue, the group definition should not be part of the cluster configuration:</p>
  <p><a href="https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165" rel="nofollow noreferrer">https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165</a></p>
  <p>(Edit: I just realized that this was probably an issue you created)</p>
  <p>Groups should be defined in the snakefile, in the workflow config, or on the command-line.</p>
  <p>However, ensure you use a recent enough version of snakemake if you try to define groups in the workflow profile, because there was a bug that prevented this to be taken into account: <a href="https://github.com/snakemake/snakemake/issues/3039" rel="nofollow noreferrer">https://github.com/snakemake/snakemake/issues/3039</a></p>
  <p>This was fixed in version 8.22 (released last month): <a href="https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0" rel="nofollow noreferrer">https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-22T12:49:46.403Z
Answers.LastActivityDate: 2024-11-22T12:49:46.403Z
Answers.UserDisplayName: bli
Body: <p>I have been trying to use Snakemake on the Dutch National Super Computer, Snellius, which is just a large Slurm cluster.</p>
  <p>On Snellius the minimum amount of cores you can use is 16 per node. So in order to make efficient use of its resources I want to use Snakemake Job Grouping. So far I have not been able to get it to work, but I admit I also don't have enough knowledge of slurm to determine if everything worked as planned...</p>
  <p>If I understand the documentation right (but I don't feel I do), I can just assign rules to a group, and they will be executed on 1 node, together. I also understand that you cannot randomly group stuff when they are far apart in the DAG. I tried grouping for fastqc using the following cluster configuration file:</p>
  <pre class="lang-yaml prettyprint-override"><code>executor: slurm
  default-resources:
    slurm_partition: &quot;rome&quot; # Default partition
    mem_mb: 1000 # By default jobs get 1000 MB of memory
    runtime: 60 # All tasks killed by default after 1 hour
    slurm_extra: &quot;'-o cluster_outputs/%j.out -e cluster_outputs/%j.err'&quot;
  printshellcmds: True
  jobs: 100
  restart-times: 3 # Re-try failed jobs 3 times, especially handy for SRA downloads which may fail randomly
  latency-wait: 60 # Wait for 60 secs before deciding a file is missing (perhaps there is filesystem latency)
  keep-going: True # Even if one process fails, keep going with the rest of the independent tasks
  rerun-incomplete: True # If a task didn't finish, re-run on the next restart
  use-conda: True # Build all env automatically
  conda-prefix: /home/fvhemert/projects/snaqs_files/snakemake_envs # Store the envs centrally
  group-components: fastqc_group=16 # Not sure what this means
  
  # rule specific resources
  set-resources:
    download_sra_files:
      threads: 1
      runtime: 120
    sra2fastq:
      threads: 6
      runtime: 360
    fastqc:
      threads: 1
      group: fastqc_group
    trim_galore:
      threads: 4
      runtime: 720
      mem_mb: 20000
    complexity_20mer_counter:
      mem_mb: 10000
    salmon:
      threads: 16
      mem_mb: 20000 # This seems to work, but can we do with less?
      runtime: 600
      group: &quot;'group_{sample}'&quot;
    tpm4_normalization:
      mem_mb: 10000 # How much do we really need?
      runtime: 30
      group: &quot;'group_{sample}'&quot;
    customqc_parameters:
      mem_mb: 10000 # How much do we really need?
      runtime: 30
      group: &quot;'group_{sample}'&quot;
  </code></pre>
  <p>In the snakefile I have added this line (but I wonder if this is necessary!):</p>
  <pre class="lang-py prettyprint-override"><code>groups = [&quot;fastqc_group&quot;, &quot;group_{sample}&quot;]
  </code></pre>
  <p>I think there is another use of Job Grouping, where can group all tasks per sample so that I can make sure that they run on the same node and can access the same local fast SSD storage. That is what I have tried to implement with the group_{sample}.</p>
  <p>If I understand correctly, the way to check if job grouping is actually working is to do a dry run and see if you actually create less jobs, for example in the overview below there should be 1 fastqc job to be submitted (which would then run all 4 in parallel). This has never happened for me, the dry run always gives this overview:</p>
  <pre class="lang-bash prettyprint-override"><code>...
  job                           count
  --------------------------  -------
  all                               1
  complexity_20mer_counter          4
  create_flagged_sampletable        1
  create_pcs_raw_files              2
  customqc_parameters               2
  customqc_report                   1
  fastqc                            4
  qc_flagging                       2
  rnaseq_multiqc                    1
  salmon                            2
  seqrun_expression_reports         1
  tpm4_normalization                2
  trim_galore                       2
  total                            25
  
  Reasons:
      (check individual jobs above for details)
      input files updated by another job:
          all, complexity_20mer_counter, create_flagged_sampletable, create_pcs_raw_files, customqc_parameters, customqc_report, qc_flagging, rnaseq_multiqc, salmon, seqrun_expression_reports, tpm4_normalization
      output files have to be generated:
          trim_galore
      updated input files:
          fastqc, trim_galore
  
  This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
  </code></pre>
  <p>Starting the workflow you can see that the jobs get a &quot;group=fastqc_group&quot; parameter:</p>
  <pre class="lang-bash prettyprint-override"><code>[Sun Oct 27 22:45:27 2024]
  rule fastqc:
      input: fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz
      output: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
      jobid: 17
      benchmark: benchmark/0053_P2017BB3S19R_S1_R2.fastqc.fastqc.benchmark.tsv
      reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
      wildcards: sample=0053_P2017BB3S19R_S1, read=R2
      resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=&lt;TBD&gt;, slurm_partition=rome, runtime=60, slurm_extra=-o cluster_outputs/%j.out -e cluster_outputs/%j.err, threads=1, group=fastqc_group
  
  
          fastqc fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz --outdir=qc/fastqc
          
  Job 17 has been submitted with SLURM jobid 8281752 (log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log).
  </code></pre>
  <p>The workflow finishes successfully but there is no sign of any job grouping... But perhaps I'm just missing it? I also have no idea what it should look like, perhaps the cluster output files contain the logs of all the grouped jobs, all grouped jobs should have a single JobID, right?? Something like that? Any pointers are welcome.</p>
  <p>If anyone could point me to some functional examples next to the documentation that would be very handy. If I get this to work I will certainly post my workflow + configs somewhere as an example.</p>
  
Tags: <snakemake><slurm>
Title: Snakemake on Slurm and job grouping
ThreadText: # Snakemake on Slurm and job grouping
  
  (Asked by: username_0 on 2024-10-27)
  
  I have been trying to use Snakemake on the Dutch National Super Computer, Snellius, which is just a large Slurm cluster.
  
  On Snellius the minimum amount of cores you can use is 16 per node. So in order to make efficient use of its resources I want to use Snakemake Job Grouping. So far I have not been able to get it to work, but I admit I also don't have enough knowledge of slurm to determine if everything worked as planned...
  
  If I understand the documentation right (but I don't feel I do), I can just assign rules to a group, and they will be executed on 1 node, together. I also understand that you cannot randomly group stuff when they are far apart in the DAG. I tried grouping for fastqc using the following cluster configuration file:
  
  ```yaml
  executor: slurm
  default-resources:
    slurm_partition: "rome" # Default partition
    mem_mb: 1000 # By default jobs get 1000 MB of memory
    runtime: 60 # All tasks killed by default after 1 hour
    slurm_extra: "'-o cluster_outputs/%j.out -e cluster_outputs/%j.err'"
  printshellcmds: True
  jobs: 100
  restart-times: 3 # Re-try failed jobs 3 times, especially handy for SRA downloads which may fail randomly
  latency-wait: 60 # Wait for 60 secs before deciding a file is missing (perhaps there is filesystem latency)
  keep-going: True # Even if one process fails, keep going with the rest of the independent tasks
  rerun-incomplete: True # If a task didn't finish, re-run on the next restart
  use-conda: True # Build all env automatically
  conda-prefix: /home/fvhemert/projects/snaqs_files/snakemake_envs # Store the envs centrally
  group-components: fastqc_group=16 # Not sure what this means
  
  # rule specific resources
  set-resources:
    download_sra_files:
      threads: 1
      runtime: 120
    sra2fastq:
      threads: 6
      runtime: 360
    fastqc:
      threads: 1
      group: fastqc_group
    trim_galore:
      threads: 4
      runtime: 720
      mem_mb: 20000
    complexity_20mer_counter:
      mem_mb: 10000
    salmon:
      threads: 16
      mem_mb: 20000 # This seems to work, but can we do with less?
      runtime: 600
      group: "'group_{sample}'"
    tpm4_normalization:
      mem_mb: 10000 # How much do we really need?
      runtime: 30
      group: "'group_{sample}'"
    customqc_parameters:
      mem_mb: 10000 # How much do we really need?
      runtime: 30
      group: "'group_{sample}'"
  
  ```
  
  In the snakefile I have added this line (but I wonder if this is necessary!):
  
  ```py
  groups = ["fastqc_group", "group_{sample}"]
  
  ```
  
  I think there is another use of Job Grouping, where can group all tasks per sample so that I can make sure that they run on the same node and can access the same local fast SSD storage. That is what I have tried to implement with the group\_{sample}.
  
  If I understand correctly, the way to check if job grouping is actually working is to do a dry run and see if you actually create less jobs, for example in the overview below there should be 1 fastqc job to be submitted (which would then run all 4 in parallel). This has never happened for me, the dry run always gives this overview:
  
  ```bash
  ...
  job                           count
  --------------------------  -------
  all                               1
  complexity_20mer_counter          4
  create_flagged_sampletable        1
  create_pcs_raw_files              2
  customqc_parameters               2
  customqc_report                   1
  fastqc                            4
  qc_flagging                       2
  rnaseq_multiqc                    1
  salmon                            2
  seqrun_expression_reports         1
  tpm4_normalization                2
  trim_galore                       2
  total                            25
  
  Reasons:
      (check individual jobs above for details)
      input files updated by another job:
          all, complexity_20mer_counter, create_flagged_sampletable, create_pcs_raw_files, customqc_parameters, customqc_report, qc_flagging, rnaseq_multiqc, salmon, seqrun_expression_reports, tpm4_normalization
      output files have to be generated:
          trim_galore
      updated input files:
          fastqc, trim_galore
  
  This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
  
  ```
  
  Starting the workflow you can see that the jobs get a "group=fastqc\_group" parameter:
  
  ```bash
  [Sun Oct 27 22:45:27 2024]
  rule fastqc:
      input: fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz
      output: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
      jobid: 17
      benchmark: benchmark/0053_P2017BB3S19R_S1_R2.fastqc.fastqc.benchmark.tsv
      reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
      wildcards: sample=0053_P2017BB3S19R_S1, read=R2
      resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=rome, runtime=60, slurm_extra=-o cluster_outputs/%j.out -e cluster_outputs/%j.err, threads=1, group=fastqc_group
  
          fastqc fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz --outdir=qc/fastqc
          
  Job 17 has been submitted with SLURM jobid 8281752 (log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log).
  
  ```
  
  The workflow finishes successfully but there is no sign of any job grouping... But perhaps I'm just missing it? I also have no idea what it should look like, perhaps the cluster output files contain the logs of all the grouped jobs, all grouped jobs should have a single JobID, right?? Something like that? Any pointers are welcome.
  
  If anyone could point me to some functional examples next to the documentation that would be very handy. If I get this to work I will certainly post my workflow + configs somewhere as an example.
  
  **Tags:** snakemake, slurm
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-11-22. Score: 2 | Accepted answer
  
  According to this answer to an issue, the group definition should not be part of the cluster configuration:
  
  <https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165>
  
  (Edit: I just realized that this was probably an issue you created)
  
  Groups should be defined in the snakefile, in the workflow config, or on the command-line.
  
  However, ensure you use a recent enough version of snakemake if you try to define groups in the workflow profile, because there was a bug that prevented this to be taken into account: <https://github.com/snakemake/snakemake/issues/3039>
  
  This was fixed in version 8.22 (released last month): <https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0>
  
  
  ---
  
  ### Answer by username_1 on 2024-10-30. Score: 2
  
  Got here from your slurm-excecutor slurm issue :)
  
  First thing's first, my use of snakemake is not for bioinformatics, I use it in a completely different field where we dont need that many small parallel jobs. I do use job grouping and to my understanding it should work as you intend it to.
  
  Also a disclaimer, I use snakemake 7.X. I still have not upgraded my workflow to work with 8.X, hence, I use the "old" slurm executor (the cookiecutter profile), but I hope my experience translates to 8.X if that's what you are working with.
  
  Now to your questions:
  The log file for a grouped rule should group all the small jobs submitted with the group. So in your case check:
  
  > log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm\_logs/rule\_fastqc/0053\_P2017BB3S19R\_S1\_R2/8281752.log
  
  If all the grouped rules are show in the log, then success. And yes, each group should only spawn one job submission with a unique jobID.
  It is kind of strange that the folder where the log is saved says "rule" in it, in my case logs are saved in a folder with the name of the group. This might be a difference between the profile and the executor.
  
  Something else that might be useful. After submitting the job, just spam squeue and see how your job queue looks like. Try with and without grouping, the difference should be quite clear.
  
  Creating groups with a wildcard in it is something I never tried, so can't help you with that one.
  
  Resource wise, remember you are being charged 16 cores anyways, which means you also get the memory allocation for that (By default 1.75gb per thread in a thin node). So if you have 4 rules that run in parallel and each of them needs... say 6gb, you can just specify the rule needs 4 threads and by default it will get 1.75\*4gb of RAM, and since the 4 rules "fit" in the allocated node, they should run in parallel.
  In other words, I stopped worrying about optimizing for memory, and I now "optimize" by threads (even though I know many of my rules use just 1 core), and fine-tune from there!
  
  > **Comments:**
  > **username_0**: Ok I got it to work, but only by defining goups in the rules, it doesn't work from the config yaml. I think I got the concept wrong, it seems that it only groups subsequent jobs? Can it also be used to make for example 16 small jobs be run each in 1 thread on a 16 core CPU?  
  > **username_0**: Ok, I think I have to keep in mind that you can only group tasks that are connected in the DAG, right?  
  > **username_0**: Ok,I'm running into this issue, will update here if I get it to work: https://github.com/snakemake/snakemake/issues/3165  
  
  ---
  
  