Answers.Body: <p>The solutions are either, upgrade <code>snakemake</code> or just take all wildcards out of <code>cluster.json</code> file and write them directly to snakemake execution command :</p>
  
  <pre><code>snakemake -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {"logs/cluster/{rule}.{wildcards}.out" -e "logs/cluster/{rule}.{wildcards}.err""
  </code></pre>
  
  <p>However, I ended up instead of using of json config file in writing a simple bash wrapper <code>snakemake_cluster.sh</code> that remembers the execution command for me in more readable format :</p>
  
  <pre><code>#!/usr/bin/env bash
  
  snakemake $@ -p --jobs 10 --cluster "bsub \
  -J {rule} \
  -q bgee \
  -n {threads} \
  -M {resources.M} \
  -R \"span[hosts=1]\" \
  -o logs/cluster/{rule}.{wildcards}.out \
  -e logs/cluster/{rule}.{wildcards}.err"
  </code></pre>
  
  <p>Edit : All spaces after <code>--cluster</code> command got to be escaped, otherwise it would add arguments to <code>snakemake</code> not to <code>bsub</code>.</p>
  
  <p>Edit 2 : <code>$@</code> means all arguments, therefore it can be either targets, flags or combinations (targets got to be specified first). So, <code>snakemake_cluster.sh my_file</code> builds <code>my_file</code>, <code>snakemake_cluster.sh --quiet</code> builds default target quietly, and <code>snakemake_cluster.sh my_file --quiet</code> builds <code>my_file</code> quietly.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-23T09:46:16.770Z
Answers.LastActivityDate: 2017-11-23T11:57:16.147Z
Answers.LastEditDate: 2017-11-23T11:57:16.147Z
Answers.UserDisplayName: Kamil S Jaron
Answers.Body: <p>As of the most recent snakemake (v5.10.0) the recommended way of defining resources is through a <a href="https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles" rel="nofollow noreferrer">profile</a>.  </p>
  
  <p>It seems you are trying to submit a job on an LSF cluster system - there is a snakemake <a href="https://github.com/Snakemake-Profiles/snakemake-lsf" rel="nofollow noreferrer">profile for LSF</a> to save you having to do this work yourself.</p>
  
  <p>It reduces your snakemake submission to something like</p>
  
  <pre class="lang-sh prettyprint-override"><code>snakemake --profile "lsf"
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-15T13:38:41.023Z
Answers.Text: Luckily I don't have to deal with `lsf` anymore, but thanks anyway, I will take a look at this feature.
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-14T14:03:20.063Z
Answers.LastActivityDate: 2020-03-14T14:03:20.063Z
Answers.UserDisplayName: Michael Hall
Answers.Body: <p>This feature was added in version 3.6.0. Look at <a href="http://snakemake.readthedocs.io/en/stable/project_info/history.html" rel="nofollow noreferrer">release notes</a>.</p>
  
  <p><code>{wildcards}</code> normally expands to a something that looks vaguely like a dictionary. In other words, something like <code>key1=value1,key2=value2</code>. Whether you really want to use that as a log file name is up to you.</p>
  
  <p>Let's use a very contrived example to demonstrate this. Here is the <code>Snakefile</code>:</p>
  
  <pre><code>rule all:
      input: "input1.human.txt"
  
  rule blah:
      output: "{sample}.{reference}.txt"
      shell: "touch {output}"
  </code></pre>
  
  <p>I'm using your <code>cluster.json</code>, so with that let's use a contrived cluster submission program:</p>
  
  <pre><code>snakemake --cluster-config cluster.json --cluster "echo 'rule.wildcards: {rule}.{wildcards} cluster: {cluster} -o {cluster.output} -e {cluster.error} -n {cluster.name}' &gt;&gt; log &amp;&amp; " --jobs 1
  </code></pre>
  
  <p>If you run that and look in <code>log</code>, you can see that <code>{rule}</code> is expanded as expected, <code>{wildcards}</code> is expanded to <code>sample=input1,reference=human</code>. The <code>output</code> and <code>error</code> parts from <code>cluster.json</code> are expanded correctly for me. Below are the contents of <code>log</code>:</p>
  
  <pre><code>rule.wildcards: blah.sample=input1,reference=human cluster: one_host="span[hosts=1]",output=logs/cluster/blah.sample=input1,reference=human.out,name=blah.sample=input1,reference=human,queue=bgee,error=logs/cluster/blah.sample=input1,reference=human.err -o logs/cluster/blah.sample=input1,reference=human.out -e logs/cluster/blah.sample=input1,reference=human.err -n blah.sample=input1,reference=human
  </code></pre>
  
  <p>I'm not a big fan of using <code>{wildcards}</code> in paths unless you're doing a parameter sweep, but to each their own. Note that the above results are from snakemake-4.3.1.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-22T22:51:55.457Z
Answers.LastActivityDate: 2017-11-23T09:32:34.083Z
Answers.LastEditDate: 2017-11-23T09:32:34.083Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I am trying to run <code>snakemake</code> version <code>3.5.4</code> on my cluster, where jobs are partially defined by a common template build according <a href="http://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration" rel="nofollow noreferrer">manual</a> and also defined though rule-specific parameters. But for some reason cluster configuration is not expanding wildcards (and command does). When I run</p>
  
  <pre><code>snakemake data/genome.fa.gz.bwt -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {cluster.output} -e {cluster.error}"
  </code></pre>
  
  <p><code>-J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host}</code> expands as expected. But <code>-o {cluster.output} -e {cluster.error}</code> is replaced by literally <code>logs/cluster/{rule}.{wildcards}.out</code> and <code>logs/cluster/{rule}.{wildcards}.err</code>, i.e. there is no expansion in config file (the config file is attached bellow).</p>
  
  <p>I am puzzled since exactly this lines are in the documentation. What am I doing wrong? Why <code>snakemake</code> is not expanding wildcards in the config file?</p>
  
  <p>Here is my <code>cluster.json</code></p>
  
  <pre><code>{
      "__default__" :
      {
          "queue"    : "bgee",
          "one_host" : "\"span[hosts=1]\"",
          "name"      : "{rule}.{wildcards}",
          "output"    : "logs/cluster/{rule}.{wildcards}.out",
          "error"     : "logs/cluster/{rule}.{wildcards}.err"
      },
  }
  </code></pre>
  
  <p>Here is relevant part of <code>Snakefile</code> :</p>
  
  <pre><code>rule index_reference :
          threads: 1
          resources: M=20000000
          input : "data/{reference}/genome.fa.gz"
          output : "data/{reference}/genome.fa.gz.bwt"
          shell : "scripts/index_genome.sh {input}"
  </code></pre>
  
Tags: <snakemake>
Title: How to specify resources for cluster in snakemake
ThreadText: # How to specify resources for cluster in snakemake
  
  (Asked by: username_0 on 2017-11-22)
  
  I am trying to run `snakemake` version `3.5.4` on my cluster, where jobs are partially defined by a common template build according [manual](http://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration) and also defined though rule-specific parameters. But for some reason cluster configuration is not expanding wildcards (and command does). When I run
  
  ```
  snakemake data/genome.fa.gz.bwt -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {cluster.output} -e {cluster.error}"
  
  ```
  
  `-J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host}` expands as expected. But `-o {cluster.output} -e {cluster.error}` is replaced by literally `logs/cluster/{rule}.{wildcards}.out` and `logs/cluster/{rule}.{wildcards}.err`, i.e. there is no expansion in config file (the config file is attached bellow).
  
  I am puzzled since exactly this lines are in the documentation. What am I doing wrong? Why `snakemake` is not expanding wildcards in the config file?
  
  Here is my `cluster.json`
  
  ```
  {
      "__default__" :
      {
          "queue"    : "bgee",
          "one_host" : "\"span[hosts=1]\"",
          "name"      : "{rule}.{wildcards}",
          "output"    : "logs/cluster/{rule}.{wildcards}.out",
          "error"     : "logs/cluster/{rule}.{wildcards}.err"
      },
  }
  
  ```
  
  Here is relevant part of `Snakefile` :
  
  ```
  rule index_reference :
          threads: 1
          resources: M=20000000
          input : "data/{reference}/genome.fa.gz"
          output : "data/{reference}/genome.fa.gz.bwt"
          shell : "scripts/index_genome.sh {input}"
  
  ```
  
  **Tags:** snakemake
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2017-11-22. Score: 5 | Accepted answer
  
  This feature was added in version 3.6.0. Look at [release notes](http://snakemake.readthedocs.io/en/stable/project_info/history.html).
  
  `{wildcards}` normally expands to a something that looks vaguely like a dictionary. In other words, something like `key1=value1,key2=value2`. Whether you really want to use that as a log file name is up to you.
  
  Let's use a very contrived example to demonstrate this. Here is the `Snakefile`:
  
  ```
  rule all:
      input: "input1.human.txt"
  
  rule blah:
      output: "{sample}.{reference}.txt"
      shell: "touch {output}"
  
  ```
  
  I'm using your `cluster.json`, so with that let's use a contrived cluster submission program:
  
  ```
  snakemake --cluster-config cluster.json --cluster "echo 'rule.wildcards: {rule}.{wildcards} cluster: {cluster} -o {cluster.output} -e {cluster.error} -n {cluster.name}' >> log && " --jobs 1
  
  ```
  
  If you run that and look in `log`, you can see that `{rule}` is expanded as expected, `{wildcards}` is expanded to `sample=input1,reference=human`. The `output` and `error` parts from `cluster.json` are expanded correctly for me. Below are the contents of `log`:
  
  ```
  rule.wildcards: blah.sample=input1,reference=human cluster: one_host="span[hosts=1]",output=logs/cluster/blah.sample=input1,reference=human.out,name=blah.sample=input1,reference=human,queue=bgee,error=logs/cluster/blah.sample=input1,reference=human.err -o logs/cluster/blah.sample=input1,reference=human.out -e logs/cluster/blah.sample=input1,reference=human.err -n blah.sample=input1,reference=human
  
  ```
  
  I'm not a big fan of using `{wildcards}` in paths unless you're doing a parameter sweep, but to each their own. Note that the above results are from snakemake-4.3.1.
  
  
  ---
  
  ### Answer by username_0 on 2017-11-23. Score: 3
  
  The solutions are either, upgrade `snakemake` or just take all wildcards out of `cluster.json` file and write them directly to snakemake execution command :
  
  ```
  snakemake -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {"logs/cluster/{rule}.{wildcards}.out" -e "logs/cluster/{rule}.{wildcards}.err""
  
  ```
  
  However, I ended up instead of using of json config file in writing a simple bash wrapper `snakemake_cluster.sh` that remembers the execution command for me in more readable format :
  
  ```
  #!/usr/bin/env bash
  
  snakemake $@ -p --jobs 10 --cluster "bsub \
  -J {rule} \
  -q bgee \
  -n {threads} \
  -M {resources.M} \
  -R \"span[hosts=1]\" \
  -o logs/cluster/{rule}.{wildcards}.out \
  -e logs/cluster/{rule}.{wildcards}.err"
  
  ```
  
  Edit : All spaces after `--cluster` command got to be escaped, otherwise it would add arguments to `snakemake` not to `bsub`.
  
  Edit 2 : `$@` means all arguments, therefore it can be either targets, flags or combinations (targets got to be specified first). So, `snakemake_cluster.sh my_file` builds `my_file`, `snakemake_cluster.sh --quiet` builds default target quietly, and `snakemake_cluster.sh my_file --quiet` builds `my_file` quietly.
  
  
  ---
  
  ### Answer by username_1 on 2020-03-14. Score: 1
  
  As of the most recent snakemake (v5.10.0) the recommended way of defining resources is through a [profile](https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles).
  
  It seems you are trying to submit a job on an LSF cluster system - there is a snakemake [profile for LSF](https://github.com/Snakemake-Profiles/snakemake-lsf) to save you having to do this work yourself.
  
  It reduces your snakemake submission to something like
  
  ```sh
  snakemake --profile "lsf"
  
  ```
  
  > **Comments:**
  > **username_0**: Luckily I don't have to deal with `lsf` anymore, but thanks anyway, I will take a look at this feature.  
  
  ---
  
  