Answers.Body: <p>While I agree with not submitting snakemake to the cluster, it's not a great solution to make your pilot program run interactively - automation via batch script should not be dependent on an open shell. </p>
  
  <p>How about allocating a whole node with many cores and working within the GNU parallel abilities of snakemake.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-11T15:45:13.437Z
Answers.LastActivityDate: 2020-02-11T15:45:13.437Z
Answers.UserDisplayName: Excel Que
Answers.Body: <p>You can omit <code>--nodes</code>, you need the following:</p>
  
  <pre><code>#!/bin/bash
  #SBATCH --ntasks-per-node=1
  #SBATCH -c threads_from_snakemake
  #SBATCH -p some_partition
  
  Commands go here
  </code></pre>
  
  <p>For slurm, you may want to modify my <a href="https://github.com/dpryan79/Misc/blob/master/MPIIE_internal/SlurmEasy" rel="noreferrer">SlurmEasy script</a>. We use that with snakemake all the time, usually of the form:</p>
  
  <pre><code>snakemake -c "SlurmEasy -t {threads} -n {rule} --mem-per-cpu {cluster.memory}" ...
  </code></pre>
  
  <p>You'll also want units with your memory requests, like <code>12G</code>. Also, in general it's best not to submit snakemake itself to the cluster, rather run it on an interactive node and have it submit jobs on your behalf (if the admins complain point out that snakemake is barely using any resources on the head node).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-05T07:05:38.400Z
Answers.LastActivityDate: 2018-09-05T07:05:38.400Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I do not understand how to specify correctly the parameters on a <code>SLURM</code> cluster for <code>snakemake</code> to use them. I tried submitting the following <code>SLURM</code> file, but it does not work this way and the number of cores used is only <code>1</code>, not <code>20</code>:</p>
  
  <pre><code>#!/bin/bash
  
  #SBATCH -p standard
  #SBATCH -A overall 
  #SBATCH --time=12:00:00
  #SBATCH --output=snakemake%A.out
  #SBATCH --error=snakemake%A.err
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=20
  #SBATCH --mem=120000
  
  snakemake
  </code></pre>
  
  <p>Then, I tried following <code>snakemake</code> <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html" rel="noreferrer">tutorial</a>. And created <code>cluster.json</code> based on the <code>SLURM</code> parameters that I need:</p>
  
  <pre><code>{
      "__default__" :
      {
          "A" : "overall",
          "time" : "24:00:00",
          "nodes": 1,
          "ntasks": 1,
          "cpus" : 20,
          "p" : "standard",
          "mem": 120000,
          "output": "snakemake%A.out",
          "error": "snakemake%A.err"
      }
  }
  </code></pre>
  
  <p>And ran <code>snakemake</code> inside a newly created <code>snakemake.sh</code> script:</p>
  
  <pre><code>#!/bin/bash
  
  snakemake -j 999 --cluster-config cluster.json --cluster "sbatch -A 
  {cluster.A} -p {cluster.p} \ 
  -t {cluster.time} --output {cluster.output} --error {cluster.error} -- 
  nodes {cluster.nodes} \
  --ntasks {cluster.ntasks} --cpus-per-task {cluster.cpus} --mem 
  {cluster.mem}"
  </code></pre>
  
  <p>And it is giving me an error now:</p>
  
  <blockquote>
    <p>sbatch: error: Unable to open file<br>
    /bin/sh: line 1: -t: command not found
    Error submitting jobscript (exit code 127):</p>
  </blockquote>
  
  <p>I am now completely lost at what I shoould actually do. I would prefer plain regular <code>.slurm</code> file submission, but how to make <code>snakemake</code> use them? Any suggestions would be greatly appreciated.</p>
  
  <p>I removed <code>\</code> - line separators in the <code>snakemake.sh</code> script:</p>
  
  <pre><code>#!/bin/bash
  
  snakemake -j 10 --cluster-config cluster.json --cluster "sbatch -A {cluster.A} -p {cluster.p} -t {cluster.time} --output {cluster.output} --error {cluster.error} --nodes {cluster.nodes} --ntasks {cluster.ntasks} --cpus-per-task {cluster.cpus} --mem {cluster.mem}"
  </code></pre>
  
  <p>And it started to run. It is not convenient though for me. I would rather prefer submitting just one job using <code>.slurm</code> file passing all of the parameters from <code>#SBATCH</code>. Is it possible?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-27T22:59:15.777Z
Comments.Text: What does the %A mean?
Comments.UserDisplayName: Biomagician
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-27T23:19:44.803Z
Comments.Text: username who is in charge of the cluster account computation resources, their payment. You can be this user, but it can also be that it is the other person, depending on who is paying for the cluster.
Comments.UserDisplayName: Nikita Vlasenko
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-09T00:12:32.473Z
Comments.Text: Can you use {rule} and {wildcards} in the output and error names in the slurm.json to name the output and error files of a job? so instead of "output": "snakemake%A.out", have ""output": "snakemake{rule}_{wilcards}.out"?
Comments.UserDisplayName: Biomagician
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-09T02:35:15.197Z
Comments.Text: good question. idk, but i would also want to learn that since it simplifies things
Comments.UserDisplayName: Nikita Vlasenko
Tags: <snakemake><slurm>
Title: Running SnakeMake on cluster
ThreadText: # Running SnakeMake on cluster
  
  (Asked by: username_0 on 2018-09-05)
  
  I do not understand how to specify correctly the parameters on a `SLURM` cluster for `snakemake` to use them. I tried submitting the following `SLURM` file, but it does not work this way and the number of cores used is only `1`, not `20`:
  
  ```
  #!/bin/bash
  
  #SBATCH -p standard
  #SBATCH -A overall 
  #SBATCH --time=12:00:00
  #SBATCH --output=snakemake%A.out
  #SBATCH --error=snakemake%A.err
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=20
  #SBATCH --mem=120000
  
  snakemake
  
  ```
  
  Then, I tried following `snakemake` [tutorial](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html). And created `cluster.json` based on the `SLURM` parameters that I need:
  
  ```
  {
      "__default__" :
      {
          "A" : "overall",
          "time" : "24:00:00",
          "nodes": 1,
          "ntasks": 1,
          "cpus" : 20,
          "p" : "standard",
          "mem": 120000,
          "output": "snakemake%A.out",
          "error": "snakemake%A.err"
      }
  }
  
  ```
  
  And ran `snakemake` inside a newly created `snakemake.sh` script:
  
  ```
  #!/bin/bash
  
  snakemake -j 999 --cluster-config cluster.json --cluster "sbatch -A 
  {cluster.A} -p {cluster.p} \ 
  -t {cluster.time} --output {cluster.output} --error {cluster.error} -- 
  nodes {cluster.nodes} \
  --ntasks {cluster.ntasks} --cpus-per-task {cluster.cpus} --mem 
  {cluster.mem}"
  
  ```
  
  And it is giving me an error now:
  
  > sbatch: error: Unable to open file  
  > /bin/sh: line 1: -t: command not found
  > Error submitting jobscript (exit code 127):
  
  I am now completely lost at what I shoould actually do. I would prefer plain regular `.slurm` file submission, but how to make `snakemake` use them? Any suggestions would be greatly appreciated.
  
  I removed `\` - line separators in the `snakemake.sh` script:
  
  ```
  #!/bin/bash
  
  snakemake -j 10 --cluster-config cluster.json --cluster "sbatch -A {cluster.A} -p {cluster.p} -t {cluster.time} --output {cluster.output} --error {cluster.error} --nodes {cluster.nodes} --ntasks {cluster.ntasks} --cpus-per-task {cluster.cpus} --mem {cluster.mem}"
  
  ```
  
  And it started to run. It is not convenient though for me. I would rather prefer submitting just one job using `.slurm` file passing all of the parameters from `#SBATCH`. Is it possible?
  
  **Tags:** snakemake, slurm
  
  > **Comments:**
  > **username_1**: What does the %A mean?  
  > **username_0**: username who is in charge of the cluster account computation resources, their payment. You can be this user, but it can also be that it is the other person, depending on who is paying for the cluster.  
  > **username_1**: Can you use {rule} and {wildcards} in the output and error names in the slurm.json to name the output and error files of a job? so instead of "output": "snakemake%A.out", have ""output": "snakemake{rule}_{wilcards}.out"?  
  > **username_0**: good question. idk, but i would also want to learn that since it simplifies things  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-09-05. Score: 10 | Accepted answer
  
  You can omit `--nodes`, you need the following:
  
  ```
  #!/bin/bash
  #SBATCH --ntasks-per-node=1
  #SBATCH -c threads_from_snakemake
  #SBATCH -p some_partition
  
  Commands go here
  
  ```
  
  For slurm, you may want to modify my [SlurmEasy script](https://github.com/dpryan79/Misc/blob/master/MPIIE_internal/SlurmEasy). We use that with snakemake all the time, usually of the form:
  
  ```
  snakemake -c "SlurmEasy -t {threads} -n {rule} --mem-per-cpu {cluster.memory}" ...
  
  ```
  
  You'll also want units with your memory requests, like `12G`. Also, in general it's best not to submit snakemake itself to the cluster, rather run it on an interactive node and have it submit jobs on your behalf (if the admins complain point out that snakemake is barely using any resources on the head node).
  
  
  ---
  
  ### Answer by username_2 on 2020-02-11. Score: 0
  
  While I agree with not submitting snakemake to the cluster, it's not a great solution to make your pilot program run interactively - automation via batch script should not be dependent on an open shell.
  
  How about allocating a whole node with many cores and working within the GNU parallel abilities of snakemake.
  
  
  ---
  
  