Answers.Body: <p>Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (<code>sample</code>) and then one character for the second (<code>read</code>). This will also cause the <code>_</code> to be included in the wildcard value <code>sample</code>.</p>
  
  <p>I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim_galore_pe output. Hence why the <code>_</code> is included in the wildcards there.</p>
  
  <p>You might be able to fix this, simply by adding a <code>_</code> in-between <code>{sample}</code> and <code>{readgroup}</code> in your first rule (and any other rules using this same pattern):</p>
  
  <pre><code>rule fastqc:
      input:
          os.path.join(config['Directories']['fastq_raw_dir'], '{sample}_{read}' + fastq_file_extension)
      output:
          html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
          zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
      wrapper:
          "0.49.0/bio/fastqc"
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T07:37:06.687Z
Answers.Text: I did this but Snakemake conjures up a new _ out of nowhere:
  Building DAG of jobs...
  MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk:
  Missing input files for rule trim_galore_pe:
  fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz
  fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T07:40:39.413Z
Answers.Text: But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake?
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T07:46:04.657Z
Answers.Text: Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both:
  Building DAG of jobs...
  MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk:
  Missing input files for rule trim_galore_pe:
  fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz
  fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz
  
  So it's either 0 or 2, not 1, as one would expect.
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T08:01:33.647Z
Answers.Text: So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now.
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T08:18:24.400Z
Answers.Text: @Freek Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it.
Answers.UserDisplayName: DavyCats
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T12:15:45.773Z
Answers.Text: Ok, I think I have found the reason, I was using code as well to create the samples_table from the same config and it was adding and removing the _ there as well. So this method does work, my apologies. It remains unhandy that I can't directly use wildcards together, But I could perhaps define a separator in the config file to solve this and keep flexibility..
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T14:33:17.483Z
Answers.LastActivityDate: 2020-02-21T14:33:17.483Z
Answers.UserDisplayName: DavyCats
Answers.Body: <p>I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using <code>wildcard_constraints</code>. E.g., towards the top of your Snakefile add:</p>
  
  <pre><code>wildcard_constraints:
      read= '|'.join([re.escape(x) for x in reads]),
      sample= '|'.join([re.escape(x) for x in samples])
  </code></pre>
  
  <p>(Personally, I use the <code>wildcard_constraints</code> directive as above quite liberally to constrain every wildcard)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T07:39:14.943Z
Answers.Text: Does this mess with how you can use rules.fastqc.output.html as the input for other rules? It gives me a lot of errors for a downstream step (multiqc).
Answers.UserDisplayName: Freek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T08:08:02.040Z
Answers.Text: As far as I know, this doesn't cause problems. If you don't show the error it's difficult to tell what's going on.
Answers.UserDisplayName: dariober
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T11:45:11.663Z
Answers.LastActivityDate: 2020-02-24T11:45:11.663Z
Answers.UserDisplayName: dariober
Body: <p>EDIT: I posted this as a bug, perhaps wait for the result of that? <a href="https://github.com/snakemake/snakemake/issues/240" rel="nofollow noreferrer">github</a></p>
  
  <p>All,</p>
  
  <p>I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:</p>
  
  <pre><code>samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
  reads = ['R1', 'R2']
  fastq_file_extension = _001.fastq.gz
  </code></pre>
  
  <pre><code>rule fastqc:
      input:
          os.path.join(config['Directories']['fastq_raw_dir'], '{sample}{read}' + fastq_file_extension)
      output:
          html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
          zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
      wrapper:
          "0.49.0/bio/fastqc"
  </code></pre>
  
  <p>However, during pipeline execution already I can see strange things happening, for this rule it reports:</p>
  
  <pre><code>[Fri Feb 21 12:39:57 2020]
  rule fastqc:
  input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
  output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
  jobid: 10
  wildcards: sample=0053_P2017BB3S20R_S2_R, read=1
  </code></pre>
  
  <p>As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.</p>
  
  <p>But the next rule, trim_galore_pe:</p>
  
  <pre><code>rule trim_galore_pe:
      input:
          [os.path.join(config['Directories']['fastq_raw_dir'], '{sample}' + read + fastq_file_extension) for read in reads]
      output:
          fastq_read1 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.1_val_1.fq.gz'),
          fastq_read2 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.2_val_2.fq.gz'),
          report_read1 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.1.fastq.gz_trimming_report.txt'),
          report_read2 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.2.fastq.gz_trimming_report.txt')
      params:
          extra="--retain_unpaired"
      wrapper:
          "0.49.0/bio/trim_galore/pe"
  </code></pre>
  
  <p>This goes wrong, during pipeline execution Snakemake reports:</p>
  
  <pre><code>[Fri Feb 21 12:40:37 2020]
  Error in rule trim_galore_pe:
  jobid: 7
  output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
  conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
  cluster_jobid: 11904662.osios
  Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
  For error details see the cluster log and the log files of the involved rule(s).
  </code></pre>
  
  <p>This is understandeble because the file fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz does not exist, it should be fastq_trimmed/0053_P2017BB3S20R_S2.1_val_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053_P2017BB3S20R_S2_).</p>
  
  <pre><code>[Fri Feb 21 12:39:57 2020]
  rule trim_galore_pe:
  input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
  output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
  jobid: 7
  wildcards: sample=0053_P2017BB3S20R_S2_
  </code></pre>
  
  <p>So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?</p>
  
Tags: <snakemake>
Title: Prevent Snakemake from changing wildcards: sample for every rule
ThreadText: # Prevent Snakemake from changing wildcards: sample for every rule
  
  (Asked by: username_0 on 2020-02-21)
  
  EDIT: I posted this as a bug, perhaps wait for the result of that? [github](https://github.com/snakemake/snakemake/issues/240)
  
  All,
  
  I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:
  
  ```
  samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
  reads = ['R1', 'R2']
  fastq_file_extension = _001.fastq.gz
  
  ```
  
  ```
  rule fastqc:
      input:
          os.path.join(config['Directories']['fastq_raw_dir'], '{sample}{read}' + fastq_file_extension)
      output:
          html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
          zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
      wrapper:
          "0.49.0/bio/fastqc"
  
  ```
  
  However, during pipeline execution already I can see strange things happening, for this rule it reports:
  
  ```
  [Fri Feb 21 12:39:57 2020]
  rule fastqc:
  input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
  output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
  jobid: 10
  wildcards: sample=0053_P2017BB3S20R_S2_R, read=1
  
  ```
  
  As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.
  
  But the next rule, trim\_galore\_pe:
  
  ```
  rule trim_galore_pe:
      input:
          [os.path.join(config['Directories']['fastq_raw_dir'], '{sample}' + read + fastq_file_extension) for read in reads]
      output:
          fastq_read1 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.1_val_1.fq.gz'),
          fastq_read2 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.2_val_2.fq.gz'),
          report_read1 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.1.fastq.gz_trimming_report.txt'),
          report_read2 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.2.fastq.gz_trimming_report.txt')
      params:
          extra="--retain_unpaired"
      wrapper:
          "0.49.0/bio/trim_galore/pe"
  
  ```
  
  This goes wrong, during pipeline execution Snakemake reports:
  
  ```
  [Fri Feb 21 12:40:37 2020]
  Error in rule trim_galore_pe:
  jobid: 7
  output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
  conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
  cluster_jobid: 11904662.osios
  Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
  For error details see the cluster log and the log files of the involved rule(s).
  
  ```
  
  This is understandeble because the file fastq\_trimmed/0053\_P2017BB3S20R\_S2\_.1\_val\_1.fq.gz does not exist, it should be fastq\_trimmed/0053\_P2017BB3S20R\_S2.1\_val\_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053\_P2017BB3S20R\_S2\_).
  
  ```
  [Fri Feb 21 12:39:57 2020]
  rule trim_galore_pe:
  input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
  output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
  jobid: 7
  wildcards: sample=0053_P2017BB3S20R_S2_
  
  ```
  
  So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?
  
  **Tags:** snakemake
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-21. Score: 1
  
  Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (`sample`) and then one character for the second (`read`). This will also cause the `_` to be included in the wildcard value `sample`.
  
  I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim\_galore\_pe output. Hence why the `_` is included in the wildcards there.
  
  You might be able to fix this, simply by adding a `_` in-between `{sample}` and `{readgroup}` in your first rule (and any other rules using this same pattern):
  
  ```
  rule fastqc:
      input:
          os.path.join(config['Directories']['fastq_raw_dir'], '{sample}_{read}' + fastq_file_extension)
      output:
          html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
          zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
      wrapper:
          "0.49.0/bio/fastqc"
  
  ```
  
  > **Comments:**
  > **username_0**: I did this but Snakemake conjures up a new _ out of nowhere: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz  
  > **username_0**: But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake?  
  > **username_0**: Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz  So it's either 0 or 2, not 1, as one would expect.  
  > **username_0**: So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now.  
  > **username_1**: @username_0 Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it.  
  > **username_0**: Ok, I think I have found the reason, I was using code as well to create the samples_table from the same config and it was adding and removing the _ there as well. So this method does work, my apologies. It remains unhandy that I can't directly use wildcards together, But I could perhaps define a separator in the config file to solve this and keep flexibility..  
  
  ---
  
  ### Answer by username_2 on 2020-02-24. Score: 1
  
  I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using `wildcard_constraints`. E.g., towards the top of your Snakefile add:
  
  ```
  wildcard_constraints:
      read= '|'.join([re.escape(x) for x in reads]),
      sample= '|'.join([re.escape(x) for x in samples])
  
  ```
  
  (Personally, I use the `wildcard_constraints` directive as above quite liberally to constrain every wildcard)
  
  > **Comments:**
  > **username_0**: Does this mess with how you can use rules.fastqc.output.html as the input for other rules? It gives me a lot of errors for a downstream step (multiqc).  
  > **username_2**: As far as I know, this doesn't cause problems. If you don't show the error it's difficult to tell what's going on.  
  
  ---
  
  