Answers.Body: <p>Your loop is more complicated than you need and it won't work properly anyway since you're using the same variable (<code>sample_paths</code>) to iterate and save. Try this instead:</p>
  <pre><code>for dir in EFS/Analyses/BGC.test/*/; do ... done
  </code></pre>
  <p>Next, all you really need is the sample names. The rest can be inferred:</p>
  <pre><code>sample_names=&quot;&quot;
  for dir in EFS/Analyses/BGC.test/*/; do
      sample_names+=&quot;$(basename &quot;$dir&quot;) &quot;
  done
  ## remove the extra space added in the end
  sample_names=$(sed 's/  *$//' &lt;&lt;&lt; &quot;$sample_names&quot;)
  export sample_paths
  </code></pre>
  <p>Then, in python, you would do:</p>
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
  
  import os
  
  base_dir=&quot;EFS/Analyses/BGC.test&quot;
  for sample_name in os.environ[&quot;sample_names&quot;].split(' '):
      gbk_file = &quot;&quot;&quot;%s/%s/antismash2/%s.LargeContigs.gbk&quot;&quot;&quot; % (
          base_dir, sample_name, sample_name
      )
      tsv_input = &quot;&quot;&quot;%s/%s/Deepbgc/Deepbgc.bgc.tsv&quot;&quot;&quot; % (base_dir, sample_name)
      json_input = &quot;&quot;&quot;%s/%s//Bagel4/00.OverviewGeneTables.json&quot;&quot;&quot; % (base_dir, sample_name)
      print(&quot;============== Sample Name:&quot;,sample_name, &quot;=============&quot;)
      print(&quot;GenBank file:&quot;, gbk_file)
      print(&quot;Tsv file:&quot;, tsv_input)
      print(&quot;Tsv file:&quot;, json_input)
  </code></pre>
  <p>I created the following test directories:</p>
  <pre><code>$ tree
  .
  └── EFS
      └── Analyses
          └── BGC.test
              ├── Sample_1
              │   ├── antismash2
              │   │   └── sample.LargeContigs.gbk
              │   ├── Bagel4
              │   │   └── 00.OverviewGeneTables.json
              │   └── Deepbgc
              │       └── Deepbgc.bgc.tsv
              ├── Sample_2
              │   ├── antismash2
              │   │   └── sample.LargeContigs.gbk
              │   ├── Bagel4
              │   │   └── 00.OverviewGeneTables.json
              │   └── Deepbgc
              │       └── Deepbgc.bgc.tsv
              └── Sample_3
                  ├── antismash2
                  │   └── sample.LargeContigs.gbk
                  ├── Bagel4
                  │   └── 00.OverviewGeneTables.json
                  └── Deepbgc
                      └── Deepbgc.bgc.tsv
  </code></pre>
  <p>And running the two snippets of code above produces:</p>
  <pre><code>$ foo.py
  ============== Sample Name: Sample_1 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_1/antismash2/Sample_1.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_1/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_1//Bagel4/00.OverviewGeneTables.json
  ============== Sample Name: Sample_2 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_2/antismash2/Sample_2.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_2/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_2//Bagel4/00.OverviewGeneTables.json
  ============== Sample Name: Sample_3 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_3/antismash2/Sample_3.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_3/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_3//Bagel4/00.OverviewGeneTables.json
  </code></pre>
  <p>All this said, I don't really understand why you are using the shell at all. You can implement all of this directly in python:</p>
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
  
  import os
  
  base_dir=&quot;EFS/Analyses/BGC.test&quot;
  sample_names = os.listdir('EFS/Analyses/BGC.test/')
  
  for sample_name in sample_names:
      gbk_file = &quot;&quot;&quot;%s/%s/antismash2/%s.LargeContigs.gbk&quot;&quot;&quot; % (
          base_dir, sample_name, sample_name
      )
      tsv_input = &quot;&quot;&quot;%s/%s/Deepbgc/Deepbgc.bgc.tsv&quot;&quot;&quot; % (base_dir, sample_name)
      json_input = &quot;&quot;&quot;%s/%s//Bagel4/00.OverviewGeneTables.json&quot;&quot;&quot; % (base_dir, sample_name)
      
      # do something with these files here
   
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T18:18:07.573Z
Answers.Text: thanks for the reply. You're right, a direct python implementation would be much easier. I was trying to familiarize myself some more with utilizing bash as well. That being said, I will try implementing the python loop you provided. Now if I have extraneous files in the main BGC.test directory, I will have to ignore these manually?
Answers.UserDisplayName: Joe_G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T19:16:26.023Z
Answers.Text: @Joe_G no, only if you have extraneous directories. The trailing slash in `for dir in EFS/Analyses/BGC.test/*/` limits the results to directories only. You can do the whole thing in bash too, it's the mixing of the two languages that makes it more complicated.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T20:03:28.870Z
Answers.Text: Gotcha, so If I were implementing directly through python (ie. negating bash all together), does that sample_names = os.listdir ... you provided in that example python code accomplish essentially the same thing? (ignoring files and looking just at directory paths)
Answers.UserDisplayName: Joe_G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-11T09:40:03.747Z
Answers.Text: @Joe_G no, that doesn't ignore files. But you can just test the results and skip any that are files or put the whole thing inside an `if os.isdir(sample_name_):`.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T15:57:37.620Z
Answers.LastActivityDate: 2022-10-10T15:57:37.620Z
Answers.UserDisplayName: terdon
Body: <p>I am in the process of writing up a bash script that runs three bioinformatics tools and then executes a python script. This python script compiles some of the data from the informatics tools output into a single merged output file. Now I need to run this on a large number of samples at a time so I am trying to set up the script to read through the directory path to where all the output from the tools is stored.</p>
  <p>These are the directory paths for the python script input files:</p>
  <pre><code>EFS/Analyses/BGC.test/Sample_Name/antismash2/sample.LargeContigs.gbk
  EFS/Analyses/BGC.test/Sample_Name/Bagel4/00.OverviewGeneTables.json
  EFS/Analyses/BGC.test/Sample_Name/Deepbgc/Deepbgc.bgc.tsv
  </code></pre>
  <p>Now there are multiple Sample_Name directories containing identical structures in the EFS/Analyses/BGC.test path. I would like to run the script on all of them</p>
  <p>To do this, I have written a <code>for</code> loop within the bash script as follows:</p>
  <pre><code>sample_paths=<span class="math-container">$(find $</span>EFS/Analyses/BGC.test -mindepth 1 -maxdepth 1 -type d \ )
  while read sample_paths; 
  do
     sample=<span class="math-container">$(basename $</span>sample_paths)
     GenBank_Input=&quot;<span class="math-container">$sample/antismash2/$</span>sample.LargeContigs.gbk&quot;;
     TsvFile_Input=&quot;<span class="math-container">$sample/Deepbgc/Deepbgc.bgc.tsv";
     Json_Input="$</span>sample/Bagel4/00.OverviewGeneTables.json&quot;;
  done &lt;&lt;&lt; “$sample_paths”
  
  export GenBank_Input
  export TsvFile_Input
  export Json_Input
  </code></pre>
  <p>The goal here being to define the sample path for every sample and then for every sample, looping through to the specific directories of each input file. I then exported the variables and used os.environ from the os package to use them in the python script as such:</p>
  <pre><code>    #parse gbk input file
      for record in SeqIO.parse(os.environ[&quot;GenBank_Input&quot;], 'genbank'):
      #parse tsv input file
      file_handle = open(os.environ[&quot;TsvFile_Input&quot;])
      #parse json input file
      json_data = json.load(open(os.environ[&quot;Json_Input&quot;]))
  </code></pre>
  <p>Now the error I get when I run the script is a <code>FileNotFoundError: [Errno 2] No such file or directory: '“”/antismash2/“”.LargeContigs.gbk'</code> so the error seems pretty clear cut. The issue seems to be with the path that I defined within the while loop variables but as of right now, I have been unable to resolve. I am trying to familiarize myself more with bash scripting as I am still fairly new to it. Any guidance on how I might adjust the code to resolve this and get the script working is much appreciated, thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-10T14:50:08.437Z
Comments.Text: Could you kindly print the output of `sample_paths` @Joe_G?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-10T14:51:44.047Z
Comments.Text: What is `$EFS`? How and where is that defined? Also, note that you have `“$sample_paths”` (curly, UTF quotes) instead of `"$sample_paths"` (normal ASCII quotes). Finally, are you expecting these variables to be strings or arrays? Bash strings interpreted as python lists? Something else?
Comments.UserDisplayName: terdon
Tags: <python><bash>
Title: Running a Python Script from a bash script on multiple input files across different directory paths
ThreadText: # Running a Python Script from a bash script on multiple input files across different directory paths
  
  (Asked by: username_0 on 2022-10-10)
  
  I am in the process of writing up a bash script that runs three bioinformatics tools and then executes a python script. This python script compiles some of the data from the informatics tools output into a single merged output file. Now I need to run this on a large number of samples at a time so I am trying to set up the script to read through the directory path to where all the output from the tools is stored.
  
  These are the directory paths for the python script input files:
  
  ```
  EFS/Analyses/BGC.test/Sample_Name/antismash2/sample.LargeContigs.gbk
  EFS/Analyses/BGC.test/Sample_Name/Bagel4/00.OverviewGeneTables.json
  EFS/Analyses/BGC.test/Sample_Name/Deepbgc/Deepbgc.bgc.tsv
  
  ```
  
  Now there are multiple Sample\_Name directories containing identical structures in the EFS/Analyses/BGC.test path. I would like to run the script on all of them
  
  To do this, I have written a `for` loop within the bash script as follows:
  
  ```
  sample_paths=$(find $EFS/Analyses/BGC.test -mindepth 1 -maxdepth 1 -type d \ )
  while read sample_paths; 
  do
     sample=$(basename $sample_paths)
     GenBank_Input="$sample/antismash2/$sample.LargeContigs.gbk";
     TsvFile_Input="$sample/Deepbgc/Deepbgc.bgc.tsv";
     Json_Input="$sample/Bagel4/00.OverviewGeneTables.json";
  done <<< “$sample_paths”
  
  export GenBank_Input
  export TsvFile_Input
  export Json_Input
  
  ```
  
  The goal here being to define the sample path for every sample and then for every sample, looping through to the specific directories of each input file. I then exported the variables and used os.environ from the os package to use them in the python script as such:
  
  ```
      #parse gbk input file
      for record in SeqIO.parse(os.environ["GenBank_Input"], 'genbank'):
      #parse tsv input file
      file_handle = open(os.environ["TsvFile_Input"])
      #parse json input file
      json_data = json.load(open(os.environ["Json_Input"]))
  
  ```
  
  Now the error I get when I run the script is a `FileNotFoundError: [Errno 2] No such file or directory: '“”/antismash2/“”.LargeContigs.gbk'` so the error seems pretty clear cut. The issue seems to be with the path that I defined within the while loop variables but as of right now, I have been unable to resolve. I am trying to familiarize myself more with bash scripting as I am still fairly new to it. Any guidance on how I might adjust the code to resolve this and get the script working is much appreciated, thanks!
  
  **Tags:** python, bash
  
  > **Comments:**
  > **username_1**: Could you kindly print the output of `sample_paths` @username_0?  
  > **username_2**: What is `$EFS`? How and where is that defined? Also, note that you have `“$sample_paths”` (curly, UTF quotes) instead of `"$sample_paths"` (normal ASCII quotes). Finally, are you expecting these variables to be strings or arrays? Bash strings interpreted as python lists? Something else?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-10. Score: 2 | Accepted answer
  
  Your loop is more complicated than you need and it won't work properly anyway since you're using the same variable (`sample_paths`) to iterate and save. Try this instead:
  
  ```
  for dir in EFS/Analyses/BGC.test/*/; do ... done
  
  ```
  
  Next, all you really need is the sample names. The rest can be inferred:
  
  ```
  sample_names=""
  for dir in EFS/Analyses/BGC.test/*/; do
      sample_names+="$(basename "$dir") "
  done
  ## remove the extra space added in the end
  sample_names=$(sed 's/  *$//' <<< "$sample_names")
  export sample_paths
  
  ```
  
  Then, in python, you would do:
  
  ```py
  #!/usr/bin/env python3
  
  import os
  
  base_dir="EFS/Analyses/BGC.test"
  for sample_name in os.environ["sample_names"].split(' '):
      gbk_file = """%s/%s/antismash2/%s.LargeContigs.gbk""" % (
          base_dir, sample_name, sample_name
      )
      tsv_input = """%s/%s/Deepbgc/Deepbgc.bgc.tsv""" % (base_dir, sample_name)
      json_input = """%s/%s//Bagel4/00.OverviewGeneTables.json""" % (base_dir, sample_name)
      print("============== Sample Name:",sample_name, "=============")
      print("GenBank file:", gbk_file)
      print("Tsv file:", tsv_input)
      print("Tsv file:", json_input)
  
  ```
  
  I created the following test directories:
  
  ```
  $ tree
  .
  └── EFS
      └── Analyses
          └── BGC.test
              ├── Sample_1
              │   ├── antismash2
              │   │   └── sample.LargeContigs.gbk
              │   ├── Bagel4
              │   │   └── 00.OverviewGeneTables.json
              │   └── Deepbgc
              │       └── Deepbgc.bgc.tsv
              ├── Sample_2
              │   ├── antismash2
              │   │   └── sample.LargeContigs.gbk
              │   ├── Bagel4
              │   │   └── 00.OverviewGeneTables.json
              │   └── Deepbgc
              │       └── Deepbgc.bgc.tsv
              └── Sample_3
                  ├── antismash2
                  │   └── sample.LargeContigs.gbk
                  ├── Bagel4
                  │   └── 00.OverviewGeneTables.json
                  └── Deepbgc
                      └── Deepbgc.bgc.tsv
  
  ```
  
  And running the two snippets of code above produces:
  
  ```
  $ foo.py
  ============== Sample Name: Sample_1 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_1/antismash2/Sample_1.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_1/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_1//Bagel4/00.OverviewGeneTables.json
  ============== Sample Name: Sample_2 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_2/antismash2/Sample_2.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_2/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_2//Bagel4/00.OverviewGeneTables.json
  ============== Sample Name: Sample_3 =============
  GenBank file: EFS/Analyses/BGC.test/Sample_3/antismash2/Sample_3.LargeContigs.gbk
  Tsv file: EFS/Analyses/BGC.test/Sample_3/Deepbgc/Deepbgc.bgc.tsv
  Tsv file: EFS/Analyses/BGC.test/Sample_3//Bagel4/00.OverviewGeneTables.json
  
  ```
  
  All this said, I don't really understand why you are using the shell at all. You can implement all of this directly in python:
  
  ```py
  #!/usr/bin/env python3
  
  import os
  
  base_dir="EFS/Analyses/BGC.test"
  sample_names = os.listdir('EFS/Analyses/BGC.test/')
  
  for sample_name in sample_names:
      gbk_file = """%s/%s/antismash2/%s.LargeContigs.gbk""" % (
          base_dir, sample_name, sample_name
      )
      tsv_input = """%s/%s/Deepbgc/Deepbgc.bgc.tsv""" % (base_dir, sample_name)
      json_input = """%s/%s//Bagel4/00.OverviewGeneTables.json""" % (base_dir, sample_name)
      
      # do something with these files here
   
  
  ```
  
  > **Comments:**
  > **username_0**: thanks for the reply. You're right, a direct python implementation would be much easier. I was trying to familiarize myself some more with utilizing bash as well. That being said, I will try implementing the python loop you provided. Now if I have extraneous files in the main BGC.test directory, I will have to ignore these manually?  
  > **username_2**: @username_0 no, only if you have extraneous directories. The trailing slash in `for dir in EFS/Analyses/BGC.test/*/` limits the results to directories only. You can do the whole thing in bash too, it's the mixing of the two languages that makes it more complicated.  
  > **username_0**: Gotcha, so If I were implementing directly through python (ie. negating bash all together), does that sample_names = os.listdir ... you provided in that example python code accomplish essentially the same thing? (ignoring files and looking just at directory paths)  
  > **username_2**: @username_0 no, that doesn't ignore files. But you can just test the results and skip any that are files or put the whole thing inside an `if os.isdir(sample_name_):`.  
  
  ---
  
  