Answers.Body: <p>This is a Snakemake feature: For small files (less than 100000 bytes), checksums are compared to determine if a job needs to run if the input file is newer than output file:</p>
  <p><a href="https://github.com/snakemake/snakemake/pull/1568" rel="noreferrer">https://github.com/snakemake/snakemake/pull/1568</a></p>
  <p>With a size of 99999 bytes:</p>
  <pre><code><span class="math-container">$ dd if=/dev/zero of=a bs=99999 count=1
  1+0 records in
  1+0 records out
  99999 bytes (100 kB, 98 KiB) copied, 0.000502722 s, 199 MB/s
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:38:00 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:38:00 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T143800.486966.snakemake.log
  <span class="math-container">$ dd if=/dev/zero of=a bs=99999 count=1
  1+0 records in
  1+0 records out
  99999 bytes (100 kB, 98 KiB) copied, 0.000180857 s, 553 MB/s
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Nothing to be done (all requested files are present and up to date).
  Complete log: .snakemake/log/2022-10-31T143813.285988.snakemake.log
  </code></pre>
  <p>With a size of 100000 bytes:</p>
  <pre><code><span class="math-container">$ dd if=/dev/zero of=a bs=100000 count=1
  1+0 records in
  1+0 records out
  100000 bytes (100 kB, 98 KiB) copied, 0.000173008 s, 578 MB/s
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:41:14 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:41:14 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T144114.211255.snakemake.log
  <span class="math-container">$ dd if=/dev/zero of=a bs=100000 count=1
  1+0 records in
  1+0 records out
  100000 bytes (100 kB, 98 KiB) copied, 5.35361 s, 18.7 kB/s
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:41:26 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Updated input files: a
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:41:26 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T144125.949828.snakemake.log
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T01:01:01.823Z
Answers.Text: Thanks!  Do you know if there is a another mechanism that snakemake uses for "flag files" (https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)?
Answers.UserDisplayName: Jules B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T13:01:09.253Z
Answers.Text: @JulesB Looks like they're just touched or created (see [handle_touch()](https://github.com/snakemake/snakemake/blob/v7.18.1/snakemake/dag.py#L670-L676)). Then [touch()](https://github.com/snakemake/snakemake/blob/v7.18.1/snakemake/io.py#L688-L726) basically just calls [os.utime()](https://docs.python.org/3/library/os.html#os.utime) or `touch --no-dereference` to set the file access and modified times to the current time.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T04:17:13.090Z
Answers.LastActivityDate: 2022-10-31T04:17:13.090Z
Answers.UserDisplayName: Steve
Body: <p>I'm trying to use snakemake to replace some make workflows, but I'm finding that snakemake does not use the file modification time to trigger a rule.</p>
  <p>Here is my snakefile</p>
  <pre><code>rule sample:
      input:
          &quot;a&quot;
      output:
          &quot;b&quot;
      shell:
          &quot;touch {output}&quot;
  </code></pre>
  <p>If I create <code>a</code> then run snakemake, <code>b</code> is created as expected:</p>
  <pre><code><span class="math-container">$ touch a
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job      count    min threads    max threads
  -----  -------  -------------  -------------
  sample       1              1              1
  total        1              1              1
  
  Select jobs to execute...
  
  [Sat Oct 29 14:36:43 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Sat Oct 29 14:36:43 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-29T143643.334050.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 b
  </code></pre>
  <p>Now, if I touch <code>a</code> then ask snakemake to update <code>b</code>, nothing happens:</p>
  <pre><code>$ touch a
  $ snakemake -c1 b
  Building DAG of jobs...
  Nothing to be done (all requested files are present and up to date).
  Complete log: .snakemake/log/2022-10-29T144005.589943.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 0 Oct 29 14:40 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 b
  </code></pre>
  <p>Modifying the contents of <code>a</code> triggers an update as expected:</p>
  <pre><code><span class="math-container">$ echo new &gt; a
  $</span> snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job      count    min threads    max threads
  -----  -------  -------------  -------------
  sample       1              1              1
  total        1              1              1
  
  Select jobs to execute...
  
  [Sat Oct 29 14:42:13 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Updated input files: a
      resources: tmpdir=/tmp
  
  [Sat Oct 29 14:42:13 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-29T144213.161985.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 4 Oct 29 14:42 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:42 b
  </code></pre>
  <p>I installed snakemake-minimal (version 7.17.1) from bioconda, am using python 3.9.13 from conda-forge, on a RHEL 7 system.</p>
  <pre><code>$ snakemake --version
  7.17.1
  $ python --version
  Python 3.9.13
  $ cat /etc/redhat-release
  Red Hat Enterprise Linux Server release 7.8 (Maipo)
  </code></pre>
  <p>I'm clearly misunderstanding something very basic about snakemake.  Any suggestions on how to debug this, descriptions of how to have snakemake pay attention to file modification times, or just an explanation of why snakemake behaves differently from make would be much appreciated!</p>
  <p>thanks in advance,</p>
  
Tags: <snakemake>
Title: Modification time does not trigger a snakemake rule
ThreadText: # Modification time does not trigger a snakemake rule
  
  (Asked by: username_0 on 2022-10-29)
  
  I'm trying to use snakemake to replace some make workflows, but I'm finding that snakemake does not use the file modification time to trigger a rule.
  
  Here is my snakefile
  
  ```
  rule sample:
      input:
          "a"
      output:
          "b"
      shell:
          "touch {output}"
  
  ```
  
  If I create `a` then run snakemake, `b` is created as expected:
  
  ```
  $ touch a
  $ snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job      count    min threads    max threads
  -----  -------  -------------  -------------
  sample       1              1              1
  total        1              1              1
  
  Select jobs to execute...
  
  [Sat Oct 29 14:36:43 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Sat Oct 29 14:36:43 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-29T143643.334050.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 b
  
  ```
  
  Now, if I touch `a` then ask snakemake to update `b`, nothing happens:
  
  ```
  $ touch a
  $ snakemake -c1 b
  Building DAG of jobs...
  Nothing to be done (all requested files are present and up to date).
  Complete log: .snakemake/log/2022-10-29T144005.589943.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 0 Oct 29 14:40 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:36 b
  
  ```
  
  Modifying the contents of `a` triggers an update as expected:
  
  ```
  $ echo new > a
  $ snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job      count    min threads    max threads
  -----  -------  -------------  -------------
  sample       1              1              1
  total        1              1              1
  
  Select jobs to execute...
  
  [Sat Oct 29 14:42:13 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Updated input files: a
      resources: tmpdir=/tmp
  
  [Sat Oct 29 14:42:13 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-29T144213.161985.snakemake.log
  $ ls -l a b
  -rw-r--r-- 1 ... ... 4 Oct 29 14:42 a
  -rw-r--r-- 1 ... ... 0 Oct 29 14:42 b
  
  ```
  
  I installed snakemake-minimal (version 7.17.1) from bioconda, am using python 3.9.13 from conda-forge, on a RHEL 7 system.
  
  ```
  $ snakemake --version
  7.17.1
  $ python --version
  Python 3.9.13
  $ cat /etc/redhat-release
  Red Hat Enterprise Linux Server release 7.8 (Maipo)
  
  ```
  
  I'm clearly misunderstanding something very basic about snakemake. Any suggestions on how to debug this, descriptions of how to have snakemake pay attention to file modification times, or just an explanation of why snakemake behaves differently from make would be much appreciated!
  
  thanks in advance,
  
  **Tags:** snakemake
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-31. Score: 5
  
  This is a Snakemake feature: For small files (less than 100000 bytes), checksums are compared to determine if a job needs to run if the input file is newer than output file:
  
  <https://github.com/snakemake/snakemake/pull/1568>
  
  With a size of 99999 bytes:
  
  ```
  $ dd if=/dev/zero of=a bs=99999 count=1
  1+0 records in
  1+0 records out
  99999 bytes (100 kB, 98 KiB) copied, 0.000502722 s, 199 MB/s
  $ snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:38:00 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:38:00 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T143800.486966.snakemake.log
  $ dd if=/dev/zero of=a bs=99999 count=1
  1+0 records in
  1+0 records out
  99999 bytes (100 kB, 98 KiB) copied, 0.000180857 s, 553 MB/s
  $ snakemake -c1 b
  Building DAG of jobs...
  Nothing to be done (all requested files are present and up to date).
  Complete log: .snakemake/log/2022-10-31T143813.285988.snakemake.log
  
  ```
  
  With a size of 100000 bytes:
  
  ```
  $ dd if=/dev/zero of=a bs=100000 count=1
  1+0 records in
  1+0 records out
  100000 bytes (100 kB, 98 KiB) copied, 0.000173008 s, 578 MB/s
  $ snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:41:14 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Missing output files: b
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:41:14 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T144114.211255.snakemake.log
  $ dd if=/dev/zero of=a bs=100000 count=1
  1+0 records in
  1+0 records out
  100000 bytes (100 kB, 98 KiB) copied, 5.35361 s, 18.7 kB/s
  $ snakemake -c1 b
  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job stats:
  job       count    min threads    max threads
  ------  -------  -------------  -------------
  sample        1              1              1
  total         1              1              1
  
  Select jobs to execute...
  
  [Mon Oct 31 14:41:26 2022]
  rule sample:
      input: a
      output: b
      jobid: 0
      reason: Updated input files: a
      resources: tmpdir=/tmp
  
  [Mon Oct 31 14:41:26 2022]
  Finished job 0.
  1 of 1 steps (100%) done
  Complete log: .snakemake/log/2022-10-31T144125.949828.snakemake.log
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks!  Do you know if there is a another mechanism that snakemake uses for "flag files" (https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)?  
  > **username_1**: @username_0 Looks like they're just touched or created (see [handle_touch()](https://github.com/snakemake/snakemake/blob/v7.18.1/snakemake/dag.py#L670-L676)). Then [touch()](https://github.com/snakemake/snakemake/blob/v7.18.1/snakemake/io.py#L688-L726) basically just calls [os.utime()](https://docs.python.org/3/library/os.html#os.utime) or `touch --no-dereference` to set the file access and modified times to the current time.  
  
  ---
  
  