Answers.Body: <p>After some trial and error, I fixed it with the expand function only in the rule all and removing the expand from the input in the rule happy_rtg_array (thanks to @Pallie), and by changing the output file paths:</p>
  <pre><code>SAMPLE_SET = ['12345', '56789', '54321']
  ID_SET = ['EF_S13', 'MF_S5', 'EF_S11', 'MF_S3', 'EF_S10', 'MF_S2']
  
  rule all:
      input:
          array = expand(&quot;results/{sample}/{samp_id}/summary.csv&quot;, zip, sample = SAMPLE_SET, samp_id = ID_SET)
  
  
  rule happy_rtg_array:
      &quot;&quot;&quot; Normalizes and compares vcf files with array samples&quot;&quot;&quot;
      input:
          query_vcf = &quot;_&quot;.join([&quot;WESP_run/results/variant_calling/{sample}&quot;,&quot;{samp_id}.vcf&quot;]),
          truth_vcf = os.path.join(config[&quot;truth_dir_vcf_gsa&quot;],&quot;{sample}.vcf.gz&quot;),
      output: 
          &quot;WESPValidation/results/{sample}/{samp_id}/summary.csv&quot;
  [more code]
  </code></pre>
  <p>Basically, I replaced all the underscores with slashes. For some reason these underscores are messing up the wildcards when trying to connect them with it. Maybe someone more knowledgeable knows why this is.</p>
  <pre><code>[Tue Oct  5 12:24:10 2023]
  rule happy_rtg_array:
      input: 12345_EF_S13.vcf
      output: results/12345/EF_S13/summary.csv
      log: happy_rtg_array/12345_EF_S13.log
      jobid: 6
      reason: Missing output files: WESPValidation/results/12345/EF_S13/summary.csv
      wildcards: sample=12345, samp_id=EF_S13
      threads: 10
      resources: mem_mb=10000, disk_mb=7855, tmpdir=/tmp, threads=1, runtime_min=120
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T13:19:12.597Z
Answers.Text: For the reason why, check the part about ambiguous wildcards: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards. I missed the amibguity of your wildcards when I wrote my answer :)
Answers.UserDisplayName: Pallie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T13:30:45.723Z
Answers.Text: Dank je @Pallie :-)
Answers.UserDisplayName: Dandelion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T12:28:22.310Z
Answers.LastActivityDate: 2023-10-05T12:28:22.310Z
Answers.UserDisplayName: Dandelion
Answers.Body: <p>Using <code>expand</code> in your rule's <code>input:</code>  causes snakemake to aggregate all vcfs into one list and leads to your undesired behaviour. By simply removing the <code>expand</code> calls from the <code>input:</code> you should get your desired result.</p>
  <p><a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function" rel="nofollow noreferrer">https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T12:38:06.327Z
Answers.Text: This is only partially the answer, I tried this before indeed, but still get the following wildcards "wildcards: sample=12345_EF, samp_id=S13" and they should be: "wildcards: sample=12345, samp_id=EF_S13", to get the right file name for the input "truth_vcf"
Answers.UserDisplayName: Dandelion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T11:46:22.997Z
Answers.LastActivityDate: 2023-10-04T11:46:22.997Z
Answers.UserDisplayName: Pallie
Body: <p>In Snakemake I want to use two input files for comparison. The one file has a sample name (e.g. 526846), with an experimental id (e.g. EF_S13' and 'MF_S5'). The file I want to compare it to has the same sample name only. I want to compare each sample with experiment ID (first files) to the file with corresponding sample name (second file). These files are in different directories. I thought I would be able to use two expand functions with two lists, as shown in the code below.
  Note: these are example lists and the lists in reality is muchlarger.</p>
  <p>Relevant code:</p>
  <pre><code>
  SAMPLE_SET = ['12345', '56789', '54321']
  ID_SET = ['EF_S13', 'MF_S5', 'EF_S11', 'MF_S3', 'EF_S10', 'MF_S2']
  
  rule all:
      input:
          array = expand(&quot;results/{sample}_{samp_id}_array/{sample}_{samp_id}.summary.csv&quot;, zip, sample = SAMPLE_SET, samp_id = ID_SET)
  
  
  rule happy_rtg_array:
      &quot;&quot;&quot; Normalizes and compares vcf files with array samples&quot;&quot;&quot;
      input:
          query_vcf = expand(&quot;WESP_run/results/variant_calling/{sample}_{samp_id}.vcf&quot;, zip, sample = SAMPLE_SET, samp_id = ID_SET),
          truth_vcf = expand(os.path.join(config[&quot;truth_dir_vcf_gsa&quot;],&quot;{sample}.vcf.gz&quot;), sample = SAMPLE_SET),
      output: 
          &quot;WESPValidation/results/{sample}_{samp_id}_array/{sample}_{samp_id}.summary.csv&quot;
  [more code]
  </code></pre>
  <p>However, when I look at the output log file with --dry-run, I get as input for the job a huge list with all the inputs in one go, rather then per file/comparison, and one output file (see relevant output), and that times the amount of the (right) combinations of the file name and id (counted from the output).</p>
  <p>Also the wildcards are not what I thought they should be. I might be confused with the expand function, as I thought how I used them now should &quot;lock in&quot; the wildcards and become placeholders filled in by the lists.</p>
  <p>Relevant output:</p>
  <pre><code>[Tue Oct  3 14:31:20 2023]
  rule happy_rtg_array:
      input: 12345_EF_S13.vcf, 12345_MF_S5.vcf, 56789_EF_S11.vcf, 56789_MF_S3.vcf, 54321_EF_S10.vcf, 54321_MF_S2.vcf, 12345.vcf, 56789.vcf, 54321.vcf
      output: results/12345_EF_S13_array/12345_EF_S13.summary.csv
      log: happy_rtg_array/12345_EF_S13.log
      jobid: 6
      reason: Missing output files: WESPValidation/results/12345_EF_S13_array/12345_EF_S13.summary.csv
      wildcards: sample=12345_EF, samp_id=S13
      threads: 10
      resources: mem_mb=10000, disk_mb=7855, tmpdir=/tmp, threads=1, runtime_min=120
  </code></pre>
  <p>I doubt this is how it should be, and wonder what I am doing wrong. Is there a better/correct way to do this.</p>
  <p>Thank you in advance</p>
  
Tags: <python><snakemake>
Title: Compare files with similar names using placeholders/wildcards in Snakemake
ThreadText: # Compare files with similar names using placeholders/wildcards in Snakemake
  
  (Asked by: username_0 on 2023-10-03)
  
  In Snakemake I want to use two input files for comparison. The one file has a sample name (e.g. 526846), with an experimental id (e.g. EF\_S13' and 'MF\_S5'). The file I want to compare it to has the same sample name only. I want to compare each sample with experiment ID (first files) to the file with corresponding sample name (second file). These files are in different directories. I thought I would be able to use two expand functions with two lists, as shown in the code below.
  Note: these are example lists and the lists in reality is muchlarger.
  
  Relevant code:
  
  ```
  
  SAMPLE_SET = ['12345', '56789', '54321']
  ID_SET = ['EF_S13', 'MF_S5', 'EF_S11', 'MF_S3', 'EF_S10', 'MF_S2']
  
  rule all:
      input:
          array = expand("results/{sample}_{samp_id}_array/{sample}_{samp_id}.summary.csv", zip, sample = SAMPLE_SET, samp_id = ID_SET)
  
  rule happy_rtg_array:
      """ Normalizes and compares vcf files with array samples"""
      input:
          query_vcf = expand("WESP_run/results/variant_calling/{sample}_{samp_id}.vcf", zip, sample = SAMPLE_SET, samp_id = ID_SET),
          truth_vcf = expand(os.path.join(config["truth_dir_vcf_gsa"],"{sample}.vcf.gz"), sample = SAMPLE_SET),
      output: 
          "WESPValidation/results/{sample}_{samp_id}_array/{sample}_{samp_id}.summary.csv"
  [more code]
  
  ```
  
  However, when I look at the output log file with --dry-run, I get as input for the job a huge list with all the inputs in one go, rather then per file/comparison, and one output file (see relevant output), and that times the amount of the (right) combinations of the file name and id (counted from the output).
  
  Also the wildcards are not what I thought they should be. I might be confused with the expand function, as I thought how I used them now should "lock in" the wildcards and become placeholders filled in by the lists.
  
  Relevant output:
  
  ```
  [Tue Oct  3 14:31:20 2023]
  rule happy_rtg_array:
      input: 12345_EF_S13.vcf, 12345_MF_S5.vcf, 56789_EF_S11.vcf, 56789_MF_S3.vcf, 54321_EF_S10.vcf, 54321_MF_S2.vcf, 12345.vcf, 56789.vcf, 54321.vcf
      output: results/12345_EF_S13_array/12345_EF_S13.summary.csv
      log: happy_rtg_array/12345_EF_S13.log
      jobid: 6
      reason: Missing output files: WESPValidation/results/12345_EF_S13_array/12345_EF_S13.summary.csv
      wildcards: sample=12345_EF, samp_id=S13
      threads: 10
      resources: mem_mb=10000, disk_mb=7855, tmpdir=/tmp, threads=1, runtime_min=120
  
  ```
  
  I doubt this is how it should be, and wonder what I am doing wrong. Is there a better/correct way to do this.
  
  Thank you in advance
  
  **Tags:** python, snakemake
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-10-05. Score: 0 | Accepted answer
  
  After some trial and error, I fixed it with the expand function only in the rule all and removing the expand from the input in the rule happy\_rtg\_array (thanks to @username_1), and by changing the output file paths:
  
  ```
  SAMPLE_SET = ['12345', '56789', '54321']
  ID_SET = ['EF_S13', 'MF_S5', 'EF_S11', 'MF_S3', 'EF_S10', 'MF_S2']
  
  rule all:
      input:
          array = expand("results/{sample}/{samp_id}/summary.csv", zip, sample = SAMPLE_SET, samp_id = ID_SET)
  
  rule happy_rtg_array:
      """ Normalizes and compares vcf files with array samples"""
      input:
          query_vcf = "_".join(["WESP_run/results/variant_calling/{sample}","{samp_id}.vcf"]),
          truth_vcf = os.path.join(config["truth_dir_vcf_gsa"],"{sample}.vcf.gz"),
      output: 
          "WESPValidation/results/{sample}/{samp_id}/summary.csv"
  [more code]
  
  ```
  
  Basically, I replaced all the underscores with slashes. For some reason these underscores are messing up the wildcards when trying to connect them with it. Maybe someone more knowledgeable knows why this is.
  
  ```
  [Tue Oct  5 12:24:10 2023]
  rule happy_rtg_array:
      input: 12345_EF_S13.vcf
      output: results/12345/EF_S13/summary.csv
      log: happy_rtg_array/12345_EF_S13.log
      jobid: 6
      reason: Missing output files: WESPValidation/results/12345/EF_S13/summary.csv
      wildcards: sample=12345, samp_id=EF_S13
      threads: 10
      resources: mem_mb=10000, disk_mb=7855, tmpdir=/tmp, threads=1, runtime_min=120
  
  ```
  
  > **Comments:**
  > **username_1**: For the reason why, check the part about ambiguous wildcards: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards. I missed the amibguity of your wildcards when I wrote my answer :)  
  > **username_0**: Dank je @username_1 :-)  
  
  ---
  
  ### Answer by username_1 on 2023-10-04. Score: 1
  
  Using `expand` in your rule's `input:` causes snakemake to aggregate all vcfs into one list and leads to your undesired behaviour. By simply removing the `expand` calls from the `input:` you should get your desired result.
  
  <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function>
  
  > **Comments:**
  > **username_0**: This is only partially the answer, I tried this before indeed, but still get the following wildcards "wildcards: sample=12345_EF, samp_id=S13" and they should be: "wildcards: sample=12345, samp_id=EF_S13", to get the right file name for the input "truth_vcf"  
  
  ---
  
  