Answers.Body: <p>Using pointers provided by @bli, I made a few changes and the Snakefile started working as expected.</p>
  <p>Instead of using <code>expand</code> for the rule's input, I switched to a function that accepts the wildcard sample as input and returns a dict with the actual inputs.</p>
  <pre><code> # Could be a one-liner but I wrote it in a verbose
   # manner so it's easier to read
   def get_input_files(in_sample_id):
       fq1 = samples.loc[str(in_sample_id), 'FQ1_id']
       fq2 = samples.loc[str(in_sample_id), 'FQ2_id']
       logdir = &quot;{in_sample}.logs&quot;.format(in_sample=str(in_sample_id))
       return_dict = { 'fq1' : fq1, 'fq2' : fq2, 'logdir' : logdir }
       return(return_dict)
  </code></pre>
  <p>And the rule <code>my_job</code> is now changed (just the <code>input</code> part):</p>
  <pre><code>rule my_job:
      input:
          unpack(get_input_files)
      output:
          &quot;{sample}_1.fastq.gz&quot;,
          &quot;{sample}_2.fastq.gz&quot;,
  ...
  ...
  </code></pre>
  <p>Hope this helps someone looking for a similar solution in the future!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-28T20:09:36.070Z
Answers.LastActivityDate: 2021-12-28T20:09:36.070Z
Answers.UserDisplayName: Ram RS
Body: <p>I'm trying to run a command using snakemake and a TSV config file. Here is my sample config file:</p>
  <pre><code>sample_id       FQ1_id  FQ2_id
  MyCleanName-1       MyFqName-1_R1.fastq.gz        MyFqName-1_R2.fastq.gz
  MyCleanName-2       MyFqName-2_R1.fastq.gz        MyFqName-2_R2.fastq.gz
  </code></pre>
  <p>Here is my snakefile:</p>
  <pre><code>import pandas as pd
  samples = pd.read_table(&quot;samples.tsv&quot;).set_index(&quot;sample_id&quot;, drop = False)
  
  rule all:
      input:
          expand(&quot;{sample}_{num}.fastq.gz&quot;, sample = samples.sample_id, num = [1,2])
  
  rule my_job:
      input:
          fq1 = expand(&quot;{fq1_id}&quot;, fq1_id = samples.FQ1_id),
          fq2 = expand(&quot;{fq2_id}&quot;, fq2_id = samples.FQ2_id),
          logdir = expand(&quot;{sample}.logs&quot;, sample = samples.sample_id)
      output:
          &quot;{sample}_1.fastq.gz&quot;,
          &quot;{sample}_2.fastq.gz&quot;,
      message:
          &quot;Running MyCmd on {wildcards.sample}&quot;
      shell:
          &quot;&quot;&quot;
          MyCmd --sample-name {wildcards.sample} -i1 {input.fq1} -i2 {input.fq2}
          &quot;&quot;&quot;
  
  
  rule create_logdir:
      output:
          directory(&quot;{sample}.logs&quot;)
      message:
          &quot;Creating log directories&quot;
      shell:
          &quot;mkdir -pv {wildcards.sample}.logs/{{MyCmd,MyOtherCmd}}&quot;
  </code></pre>
  <p>When I run <code>snakemake -np MyCleanName-1.fastq.gz</code>, I get the following:</p>
  <pre><code>Building DAG of jobs...
  Job counts:
          count   jobs
          2       create_logdir
          1       my_job
          3
  
  [Fri Dec 24 16:38:54 2021]
  Job 1: Creating log directories
  
  mkdir -pv MyCleanName-1.logs/{MyCmd,MyOtherCmd}
  
  [Fri Dec 24 16:38:54 2021]
  Job 2: Creating log directories
  
  mkdir -pv MyCleanName-2.logs/{MyCmd,MyOtherCmd}
  
  [Fri Dec 24 16:38:54 2021]
  Job 0: Running MyCmd on MDA-IBC-3
  
  
          MyCmd         --sample-name MyCleanName-1 -i1 MyFqName-1_R1.fastq.gz MyFqName-2_R1.fastq.gz -i2 MyFqName-1_R2.fastq.gz MyFqName-2_R2.fastq.gz
  
  Job counts:
          count   jobs
          2       create_logdir
          1       my_job
          3
  This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
  </code></pre>
  <p>How can I restrict this so the run only creates log files for the Clean Name being used and uses only the corresponding FASTQs? I feel like I'm missing something trivial here. Should I not be using the config file variable <code>samples</code> directly in the rule <code>my_job</code>, but somehow wiring up the corresponding <code>sample_id</code> and <code>FQ_id</code>s on the <code>all</code> rule?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-26T09:41:21.933Z
Comments.Text: `expand` is a Snakemake help function that tends to create confusion for beginners. The goal is to generate lists of file names from patterns, usually to generate the input of a "gathering" / "summarizing" rule. If you want a single file name, I would recommend to use standard Python (>=3.6) f-strings.
Comments.UserDisplayName: bli
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-26T09:45:57.040Z
Comments.Text: However, in your case, I think that the use of a function as input will likely be part of the solution: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?#functions-as-input-files Maybe you can have a look at this answer for some ideas of how to proceed: https://stackoverflow.com/a/64660447/1878788
Comments.UserDisplayName: bli
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-27T15:38:16.923Z
Comments.Text: Thank you, @bli - I'll look into this - functions as input might make the most sense for my case.
Comments.UserDisplayName: Ram RS
Tags: <snakemake>
Title: Pick matching entry from snakemake config table
ThreadText: # Pick matching entry from snakemake config table
  
  (Asked by: username_0 on 2021-12-24)
  
  I'm trying to run a command using snakemake and a TSV config file. Here is my sample config file:
  
  ```
  sample_id       FQ1_id  FQ2_id
  MyCleanName-1       MyFqName-1_R1.fastq.gz        MyFqName-1_R2.fastq.gz
  MyCleanName-2       MyFqName-2_R1.fastq.gz        MyFqName-2_R2.fastq.gz
  
  ```
  
  Here is my snakefile:
  
  ```
  import pandas as pd
  samples = pd.read_table("samples.tsv").set_index("sample_id", drop = False)
  
  rule all:
      input:
          expand("{sample}_{num}.fastq.gz", sample = samples.sample_id, num = [1,2])
  
  rule my_job:
      input:
          fq1 = expand("{fq1_id}", fq1_id = samples.FQ1_id),
          fq2 = expand("{fq2_id}", fq2_id = samples.FQ2_id),
          logdir = expand("{sample}.logs", sample = samples.sample_id)
      output:
          "{sample}_1.fastq.gz",
          "{sample}_2.fastq.gz",
      message:
          "Running MyCmd on {wildcards.sample}"
      shell:
          """
          MyCmd --sample-name {wildcards.sample} -i1 {input.fq1} -i2 {input.fq2}
          """
  
  rule create_logdir:
      output:
          directory("{sample}.logs")
      message:
          "Creating log directories"
      shell:
          "mkdir -pv {wildcards.sample}.logs/{{MyCmd,MyOtherCmd}}"
  
  ```
  
  When I run `snakemake -np MyCleanName-1.fastq.gz`, I get the following:
  
  ```
  Building DAG of jobs...
  Job counts:
          count   jobs
          2       create_logdir
          1       my_job
          3
  
  [Fri Dec 24 16:38:54 2021]
  Job 1: Creating log directories
  
  mkdir -pv MyCleanName-1.logs/{MyCmd,MyOtherCmd}
  
  [Fri Dec 24 16:38:54 2021]
  Job 2: Creating log directories
  
  mkdir -pv MyCleanName-2.logs/{MyCmd,MyOtherCmd}
  
  [Fri Dec 24 16:38:54 2021]
  Job 0: Running MyCmd on MDA-IBC-3
  
          MyCmd         --sample-name MyCleanName-1 -i1 MyFqName-1_R1.fastq.gz MyFqName-2_R1.fastq.gz -i2 MyFqName-1_R2.fastq.gz MyFqName-2_R2.fastq.gz
  
  Job counts:
          count   jobs
          2       create_logdir
          1       my_job
          3
  This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
  
  ```
  
  How can I restrict this so the run only creates log files for the Clean Name being used and uses only the corresponding FASTQs? I feel like I'm missing something trivial here. Should I not be using the config file variable `samples` directly in the rule `my_job`, but somehow wiring up the corresponding `sample_id` and `FQ_id`s on the `all` rule?
  
  **Tags:** snakemake
  
  > **Comments:**
  > **username_1**: `expand` is a Snakemake help function that tends to create confusion for beginners. The goal is to generate lists of file names from patterns, usually to generate the input of a "gathering" / "summarizing" rule. If you want a single file name, I would recommend to use standard Python (>=3.6) f-strings.  
  > **username_1**: However, in your case, I think that the use of a function as input will likely be part of the solution: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?#functions-as-input-files Maybe you can have a look at this answer for some ideas of how to proceed: https://stackoverflow.com/a/64660447/1878788  
  > **username_0**: Thank you, @username_1 - I'll look into this - functions as input might make the most sense for my case.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-12-28. Score: 1 | Accepted answer
  
  Using pointers provided by @username_1, I made a few changes and the Snakefile started working as expected.
  
  Instead of using `expand` for the rule's input, I switched to a function that accepts the wildcard sample as input and returns a dict with the actual inputs.
  
  ```
   # Could be a one-liner but I wrote it in a verbose
   # manner so it's easier to read
   def get_input_files(in_sample_id):
       fq1 = samples.loc[str(in_sample_id), 'FQ1_id']
       fq2 = samples.loc[str(in_sample_id), 'FQ2_id']
       logdir = "{in_sample}.logs".format(in_sample=str(in_sample_id))
       return_dict = { 'fq1' : fq1, 'fq2' : fq2, 'logdir' : logdir }
       return(return_dict)
  
  ```
  
  And the rule `my_job` is now changed (just the `input` part):
  
  ```
  rule my_job:
      input:
          unpack(get_input_files)
      output:
          "{sample}_1.fastq.gz",
          "{sample}_2.fastq.gz",
  ...
  ...
  
  ```
  
  Hope this helps someone looking for a similar solution in the future!
  
  
  ---
  
  