Answers.Body: <p>I suggest using package <a href="https://docs.python.org/3/library/glob.html" rel="nofollow noreferrer">glob</a>, which supports unix-style pattern search. For instance finding all .py files in my repo:</p>
  <pre><code>import glob 
  all_python_files = glob.glob(r'c:\personal\coderepo\**\*.py', recursive=True)
  </code></pre>
  <p>Which outputs full paths:</p>
  <blockquote>
  <p>'c:\personal\coderepo\SimpleHTR\data\checkDirs.py', <br>
  'c:\personal\coderepo\....</p>
  </blockquote>
  <p>I typically like to input the glob.glob to <a href="https://pandas.pydata.org/docs/" rel="nofollow noreferrer">Pandas</a> dataframes:</p>
  <pre><code>all_files = glob.glob(f'{base_folder}/*')
  all_files = pd.DataFrame(all_files, columns=['full_path'])
  all_files['my_filename'] = all_files['full_path'].apply(lambda x: os.path.basename(x))
  
  #do whatever filtering is required:
  all_files = all_files[all_files['my_file_name'].str.contains('regex_here')]
  </code></pre>
  <p>Then perform what ever I need to do in loop:</p>
  <pre><code>for idx, _ in all_files.iterrows():
      do_something(_['full_path'])
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-02T07:09:28.767Z
Answers.LastActivityDate: 2023-10-02T07:09:28.767Z
Answers.UserDisplayName: pinegulf
Body: <p>I am a beginner in snakemake and I'm trying to develop a working flow.</p>
  <p>Imagine I have multiple folders and in each folder I have a tumour and normal sample files:</p>
  <pre><code>SRC120/ 
        SRC120-T1.fastq
        SRC120-N1.fastq
  
  SRC121/ 
        SRC121-T1.fastq
        SRC121-N1.fastq
  </code></pre>
  <p>and so on (in total I have more than 100 SRC folders).</p>
  <p>I am trying to make a snakemake rule to make a symlink of these fastq files.</p>
  <p>What I have written so far is:</p>
  <pre class="lang-py prettyprint-override"><code>import os
  
  SAMPLES=os.listdir(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;)                                               
  
  def getFastqFilesForTissu(wildcards):
      fastqs = list()
      # Beware no other files than fastqs should be there
      for s in os.listdir(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;+wildcards.sample):
          fastqs.append(os.path.join(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;,wildcards.sample,s))
      return fastqs
  
  
  rule all:                                                                        
      input:                                                                       
          expand(&quot;results/{sample}/{sample}.fastq&quot;, sample=SAMPLES)                   
  
  
  rule foo:
      input:
          fastq_files = getFastqFilesForTissu 
      output:
          &quot;results/{sample}/{sample}.fastq&quot;
      shell:
          'ln -s {input} {output}'
  </code></pre>
  <p>This works if I have one file in each folder and when the file name matches with the folder's name (i.e., when there is no T1 or N1 suffix).</p>
  <p>Is there any way I could modify my rule to get it working for all samples in SRC directories? Or a better way to handle this issue?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-15T07:46:55.483Z
Comments.Text: What is the output you expect? What values do you want your `{input}` and `{output}` to have for a given example? This looks like a simple shell operation, do you really need to use snakemake?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-15T16:54:40.903Z
Comments.Text: This is just a very small part of a big workflow. the output is symlinked fastq files in the results folder
Comments.UserDisplayName: Anna1364
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-15T16:57:01.427Z
Comments.Text: What symlinked files? What name should the symlink have and what should its target be? The more detail you give, the better the chances someone will be able to answer. Please [edit] your question and add the expected result so we know what you need.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-15T19:16:48.467Z
Comments.Text: All of these information is in the code if you read through the snakemake code. There is no need for any edition.
Comments.UserDisplayName: Anna1364
Tags: <python><snakemake>
Title: How can I develop a snakemake rule for all files in multiple directories?
ThreadText: # How can I develop a snakemake rule for all files in multiple directories?
  
  (Asked by: username_0 on 2023-09-15)
  
  I am a beginner in snakemake and I'm trying to develop a working flow.
  
  Imagine I have multiple folders and in each folder I have a tumour and normal sample files:
  
  ```
  SRC120/ 
        SRC120-T1.fastq
        SRC120-N1.fastq
  
  SRC121/ 
        SRC121-T1.fastq
        SRC121-N1.fastq
  
  ```
  
  and so on (in total I have more than 100 SRC folders).
  
  I am trying to make a snakemake rule to make a symlink of these fastq files.
  
  What I have written so far is:
  
  ```py
  import os
  
  SAMPLES=os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/")                                               
  
  def getFastqFilesForTissu(wildcards):
      fastqs = list()
      # Beware no other files than fastqs should be there
      for s in os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/"+wildcards.sample):
          fastqs.append(os.path.join("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/",wildcards.sample,s))
      return fastqs
  
  rule all:                                                                        
      input:                                                                       
          expand("results/{sample}/{sample}.fastq", sample=SAMPLES)                   
  
  rule foo:
      input:
          fastq_files = getFastqFilesForTissu 
      output:
          "results/{sample}/{sample}.fastq"
      shell:
          'ln -s {input} {output}'
  
  ```
  
  This works if I have one file in each folder and when the file name matches with the folder's name (i.e., when there is no T1 or N1 suffix).
  
  Is there any way I could modify my rule to get it working for all samples in SRC directories? Or a better way to handle this issue?
  
  **Tags:** python, snakemake
  
  > **Comments:**
  > **username_1**: What is the output you expect? What values do you want your `{input}` and `{output}` to have for a given example? This looks like a simple shell operation, do you really need to use snakemake?  
  > **username_0**: This is just a very small part of a big workflow. the output is symlinked fastq files in the results folder  
  > **username_1**: What symlinked files? What name should the symlink have and what should its target be? The more detail you give, the better the chances someone will be able to answer. Please [edit] your question and add the expected result so we know what you need.  
  > **username_0**: All of these information is in the code if you read through the snakemake code. There is no need for any edition.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-02. Score: 2
  
  I suggest using package [glob](https://docs.python.org/3/library/glob.html), which supports unix-style pattern search. For instance finding all .py files in my repo:
  
  ```
  import glob 
  all_python_files = glob.glob(r'c:\personal\coderepo\**\*.py', recursive=True)
  
  ```
  
  Which outputs full paths:
  
  > 'c:\personal\coderepo\SimpleHTR\data\checkDirs.py',   
  > 'c:\personal\coderepo\....
  
  I typically like to input the glob.glob to [Pandas](https://pandas.pydata.org/docs/) dataframes:
  
  ```
  all_files = glob.glob(f'{base_folder}/*')
  all_files = pd.DataFrame(all_files, columns=['full_path'])
  all_files['my_filename'] = all_files['full_path'].apply(lambda x: os.path.basename(x))
  
  #do whatever filtering is required:
  all_files = all_files[all_files['my_file_name'].str.contains('regex_here')]
  
  ```
  
  Then perform what ever I need to do in loop:
  
  ```
  for idx, _ in all_files.iterrows():
      do_something(_['full_path'])
  
  ```
  
  
  ---
  
  