Answers.Body: <blockquote>
  <p>it seems cumbersome to create them to ensure execution order in my case</p>
  </blockquote>
  <p>It took me a while to figure out, but it did not seem cumbersome to me.</p>
  <p>You can re-use your rules, see below, and make Step 1 of S2 depend on Step 2 of S1 like so:</p>
  <pre><code>SAMPLES = ['S1', 'S2']
  
  
  wildcard_constraints:
      sample = &quot;S[12]&quot;
  
  rule all:
    input: expand(&quot;{sample}.done&quot;, sample = SAMPLES)
  
  
  
  rule step1:
    input: data=&quot;{sample}&quot;
    output: data=&quot;{sample}.step1.done&quot;
    shell: &quot;touch {output.data}&quot;
  
  rule step2:
    input: data=&quot;{sample}.step1.done&quot;
    output: data=&quot;{sample}.done&quot;
    shell: &quot;touch {output.data}&quot;
  
  
  # step2 of S1 has the flag file as output
  
  use rule step2 as step2ForS1 with:
      input: data=&quot;S1.step1.done&quot;
      output: data=&quot;S1.done&quot;, flag=touch(&quot;flags/step2-s1-done&quot;)
  
  # step1 of S2 needs the flag file as input
  
  use rule step1 as step1ForS2 with:
      input: data=&quot;S2&quot;, flag=&quot;flags/step2-s1-done&quot;
      output: data=&quot;S2.step1.done&quot;
  
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-14T22:47:41.430Z
Answers.LastActivityDate: 2022-10-14T22:48:52.220Z
Answers.LastEditDate: 2022-10-14T22:48:52.220Z
Answers.UserDisplayName: mvtango
Answers.Body: <p>The problem that you are encountering, is that when snakemake determines the order to run a set of rules it first builds a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" rel="nofollow noreferrer">Directed Acyclic Graph</a> (DAG) of all the rules. This shows which rules are are dependant on each other, and what order they can be safely executed in, without breaking downstream dependencies. </p>
  
  <p>In general, if you want tasks to execute one after another which do not share an explicit file dependency, you can use a <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files" rel="nofollow noreferrer">Flag File</a> as an ouput for <code>rule 1</code> then reference it as an input in <code>rule 2</code>. </p>
  
  <p>Unfortunately, I do not think this will help you in your case. You are expanding on the <code>SAMPLES</code> list and creating a wildcard. in the <code>DAG</code> this creates two competely separate execution branches which have no dependencies with each other. <code>step1</code> and <code>step2</code> are guaranteed to occur in order for each sample, however there is no guarantees between samples. </p>
  
  <p>Is there a specific reason why you need all of <code>S2</code> tasks to be run after <code>S1</code>? If so, then your best bet would be to create explicit rules for sample 1 and sample 2, using the <code>Flag File</code> to incidate when <code>step2</code> for <code>S1</code> is done.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T15:54:10.610Z
Answers.Text: In typical analysis, we would like to see some results for the first sample, and make sure it is correct. So I would like to see S1 processed all the way first. Thanks for offering the Flag File feature. But it seems cumbersome to create them to ensure execution order in my case.
Answers.UserDisplayName: zhanxw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T00:11:06.190Z
Answers.Text: @zhanxw your need seems to mostly center around QA of the pipeline, and making sure that its producing the right outputs. I often have to validate a pipeline before running 10s or 1000s of samples. The best way I found is to not try to gate the pipeline  like you are suggesting, but just run it once with a single sample,  then run it again with all of the remaining samples once you are satisfied.
Answers.UserDisplayName: Patrick Magee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T03:19:53.450Z
Answers.Text: I see your points: run one sample at a time. I can do that, but that is not convenient as I need to write another script to run snakemake on each sample. I am curious if snakemake has already some solutions. Thanks.
Answers.UserDisplayName: zhanxw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T15:24:01.677Z
Answers.LastActivityDate: 2020-05-04T15:24:01.677Z
Answers.UserDisplayName: Patrick Magee
Answers.Body: <p>I think you should use the <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#priorities" rel="noreferrer">priority</a> directive to give precedence to downstream rule(s).</p>
  
  <p>Here below I give higher priority to <code>step2</code> so when the first sample has completed <code>step1</code> snakemake will run step2 on that sample rather than submitting another sample to step1. </p>
  
  <pre><code>SAMPLES = ['S1', 'S2', 'S3']
  
  wildcard_constraints:
      sample= '|'.join([re.escape(x) for x in SAMPLES])
  
  rule all:
      input: expand("{sample}.done", sample = SAMPLES)
  
  rule step1:
      priority: 1
      output: "{sample}.step1.done"
      shell: 
          r"""
          sleep 5
          touch {output}
          """
  
  rule step2:
      priority: 10
      input: "{sample}.step1.done"
      output: "{sample}.done"
      shell: 
          r"""
          sleep 5
          touch {output}
          """
  </code></pre>
  
  <p>(Note that without <code>wildcard_constraints</code> your/this code throws <code>AmbiguousRuleException</code>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T07:53:13.990Z
Answers.LastActivityDate: 2020-05-05T07:53:13.990Z
Answers.UserDisplayName: dariober
Body: <p>My goal is to process two samples, <code>S1</code> and <code>S2</code>, using two rules, <code>step1</code>, <code>step2</code>.
  The Snakemake file is like:</p>
  
  <pre><code>SAMPLES = ['S1', 'S2']
  
  rule all:
    input: expand("{sample}.done", sample = SAMPLES)
  
  rule step1:
    input: "{sample}"
    output: "{sample}.step1.done"
    shell: "touch {output}"
  
  rule step2:
    input: "{sample}.step1.done"
    output: "{sample}.done"
    shell: "touch {output}"
  </code></pre>
  
  <p>The current execution order is: </p>
  
  <pre><code>step1 on S1
  step1 on S2
  step2 on S1
  step2 on S2
  </code></pre>
  
  <p>Is there a way to change the order to:</p>
  
  <pre><code>step1 on S1
  step2 on S1
  step1 on S2
  step2 on S2
  </code></pre>
  
  <p>Basically, I want to complete <code>S1</code>, then <code>S2</code>. 
  Is there a way to specify the order of executions in Snakemake?</p>
  
Tags: <snakemake>
Title: How to modify execution orders for Snakemake?
ThreadText: # How to modify execution orders for Snakemake?
  
  (Asked by: username_0 on 2020-05-03)
  
  My goal is to process two samples, `S1` and `S2`, using two rules, `step1`, `step2`.
  The Snakemake file is like:
  
  ```
  SAMPLES = ['S1', 'S2']
  
  rule all:
    input: expand("{sample}.done", sample = SAMPLES)
  
  rule step1:
    input: "{sample}"
    output: "{sample}.step1.done"
    shell: "touch {output}"
  
  rule step2:
    input: "{sample}.step1.done"
    output: "{sample}.done"
    shell: "touch {output}"
  
  ```
  
  The current execution order is:
  
  ```
  step1 on S1
  step1 on S2
  step2 on S1
  step2 on S2
  
  ```
  
  Is there a way to change the order to:
  
  ```
  step1 on S1
  step2 on S1
  step1 on S2
  step2 on S2
  
  ```
  
  Basically, I want to complete `S1`, then `S2`.
  Is there a way to specify the order of executions in Snakemake?
  
  **Tags:** snakemake
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2020-05-05. Score: 5 | Accepted answer
  
  I think you should use the [priority](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#priorities) directive to give precedence to downstream rule(s).
  
  Here below I give higher priority to `step2` so when the first sample has completed `step1` snakemake will run step2 on that sample rather than submitting another sample to step1.
  
  ```
  SAMPLES = ['S1', 'S2', 'S3']
  
  wildcard_constraints:
      sample= '|'.join([re.escape(x) for x in SAMPLES])
  
  rule all:
      input: expand("{sample}.done", sample = SAMPLES)
  
  rule step1:
      priority: 1
      output: "{sample}.step1.done"
      shell: 
          r"""
          sleep 5
          touch {output}
          """
  
  rule step2:
      priority: 10
      input: "{sample}.step1.done"
      output: "{sample}.done"
      shell: 
          r"""
          sleep 5
          touch {output}
          """
  
  ```
  
  (Note that without `wildcard_constraints` your/this code throws `AmbiguousRuleException`)
  
  
  ---
  
  ### Answer by username_1 on 2022-10-14. Score: 1
  
  > it seems cumbersome to create them to ensure execution order in my case
  
  It took me a while to figure out, but it did not seem cumbersome to me.
  
  You can re-use your rules, see below, and make Step 1 of S2 depend on Step 2 of S1 like so:
  
  ```
  SAMPLES = ['S1', 'S2']
  
  wildcard_constraints:
      sample = "S[12]"
  
  rule all:
    input: expand("{sample}.done", sample = SAMPLES)
  
  rule step1:
    input: data="{sample}"
    output: data="{sample}.step1.done"
    shell: "touch {output.data}"
  
  rule step2:
    input: data="{sample}.step1.done"
    output: data="{sample}.done"
    shell: "touch {output.data}"
  
  # step2 of S1 has the flag file as output
  
  use rule step2 as step2ForS1 with:
      input: data="S1.step1.done"
      output: data="S1.done", flag=touch("flags/step2-s1-done")
  
  # step1 of S2 needs the flag file as input
  
  use rule step1 as step1ForS2 with:
      input: data="S2", flag="flags/step2-s1-done"
      output: data="S2.step1.done"
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2020-05-04. Score: 0
  
  The problem that you are encountering, is that when snakemake determines the order to run a set of rules it first builds a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) of all the rules. This shows which rules are are dependant on each other, and what order they can be safely executed in, without breaking downstream dependencies.
  
  In general, if you want tasks to execute one after another which do not share an explicit file dependency, you can use a [Flag File](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files) as an ouput for `rule 1` then reference it as an input in `rule 2`.
  
  Unfortunately, I do not think this will help you in your case. You are expanding on the `SAMPLES` list and creating a wildcard. in the `DAG` this creates two competely separate execution branches which have no dependencies with each other. `step1` and `step2` are guaranteed to occur in order for each sample, however there is no guarantees between samples.
  
  Is there a specific reason why you need all of `S2` tasks to be run after `S1`? If so, then your best bet would be to create explicit rules for sample 1 and sample 2, using the `Flag File` to incidate when `step2` for `S1` is done.
  
  > **Comments:**
  > **username_0**: In typical analysis, we would like to see some results for the first sample, and make sure it is correct. So I would like to see S1 processed all the way first. Thanks for offering the Flag File feature. But it seems cumbersome to create them to ensure execution order in my case.  
  > **username_2**: @username_0 your need seems to mostly center around QA of the pipeline, and making sure that its producing the right outputs. I often have to validate a pipeline before running 10s or 1000s of samples. The best way I found is to not try to gate the pipeline  like you are suggesting, but just run it once with a single sample,  then run it again with all of the remaining samples once you are satisfied.  
  > **username_0**: I see your points: run one sample at a time. I can do that, but that is not convenient as I need to write another script to run snakemake on each sample. I am curious if snakemake has already some solutions. Thanks.  
  
  ---
  
  