Answers.Body: <blockquote>
  <p>The first two digits of vcf file names defines which sub-folder they are located.</p>
  </blockquote>
  <p>You could use this information to edit temp.txt so it contains the full path:</p>
  <pre class="lang-bash prettyprint-override"><code>grep -v &quot;.tbi$&quot; temp.txt | sed 's/^\(.\)\(.\)/\1\2\/\1\2/' &gt; new_temp.txt
  </code></pre>
  <p>I'm assuming here that you don't actually want to run <code>dx run swiss-army-knife</code> on the vcf indexes.</p>
  <p>This will turn the example of temp.txt you gave into this:</p>
  <pre><code>32/3295094_24053_0_0.dragen.hard-filtered.vcf.gz
  33/3309510_24053_0_0.dragen.hard-filtered.vcf.gz
  34/3468518_24053_0_0.dragen.hard-filtered.vcf.gz
  34/3471960_24053_0_0.dragen.hard-filtered.vcf.gz
  </code></pre>
  <p>You can then simply run a single for loop over that, instead of the nested one you are using now.</p>
  <p>eg.</p>
  <pre class="lang-bash prettyprint-override"><code>while read vcf
    do
      dx run swiss-army-knife [...]
    done &lt;  new_temp.txt
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T12:50:59.920Z
Answers.LastActivityDate: 2024-02-23T14:39:17.773Z
Answers.LastEditDate: 2024-02-23T14:39:17.773Z
Answers.UserDisplayName: DavyCats
Body: <p>Please somebody debug my code</p>
  <p>In sub-folders there are <code>500000 vcf files</code> from which I only need <code>500 vcf files</code> listed in <code>temp.txt</code> which randomly distributed in sub-folders 10 to 60. The code reads each of 500000 vcf files one by one, if a file was in temp.txt, the code runs</p>
  <pre><code>dx run swiss-army-knife -icmd=&quot;tabix -h <span class="math-container">$vcf -R $</span>REGIONS_FILE &gt; <span class="math-container">$output_file" -iin="$</span>filename_root&quot; -iin=&quot;<span class="math-container">$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$</span>OUTPUT_FOLDER&quot; --detach --priority high --yes
      done &lt; &quot;$VCF_LIST_FILE&quot;
  </code></pre>
  <p>on that if not, the code goes to next file such a process takes years. I want this code only locates files listed in temp.txt and run the main code on them by iteration.</p>
  <pre><code>#!/bin/bash
  
  # Set the working directory to the desired location within DNAnexus
  
  # Number of concurrent processes
  N=4
  
  # Iterate from 10 to 60
  for folder_number in {10..60}; do
    (
      # Define the input files and parameters for this iteration
      folder_path=&quot;/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/$folder_number&quot;
      VCF_LIST_FILE=&quot;temp.txt&quot;
      REGIONS_FILE='file.bed'
      OUTPUT_FOLDER='/out'  # Replace with your desired output folder
  
      # Set the working directory to the folder for this iteration
      dx cd &quot;$folder_path&quot;
  
      # Read each VCF file from the list
      while read -r vcf; do
        filename_root=<span class="math-container">$(basename "$</span>vcf&quot;)
        output_file=&quot;${filename_root}_extracted.vcf.gz&quot;
  
        # Use dx run swiss-army-knife to execute tabix on the VCF file
        dx run swiss-army-knife -icmd=&quot;tabix -h <span class="math-container">$vcf -R $</span>REGIONS_FILE &gt; <span class="math-container">$output_file" -iin="$</span>filename_root&quot; -iin=&quot;<span class="math-container">$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$</span>OUTPUT_FOLDER&quot; --detach --priority high --yes
      done &lt; &quot;$VCF_LIST_FILE&quot;
    ) &amp;
  
    # Limit the number of concurrent processes
    ((i=i%N)); ((i++==0)) &amp;&amp; wait
  done
  
  # Wait for all background jobs to finish
  wait
  
  # Finished message
  echo &quot;++ Finished extracting genomic coordinates from VCF files&quot;
  </code></pre>
  <p>This is how temp.txt looks. The first two digits of vcf file names defines which sub-folder they are located. For instance, the below vcf files distributed in sub-folders 32, 33 and 34</p>
  <pre><code>head temp_new.txt
  head: temp_new.txt: No such file or directory
  izadia@LV19Y7325V Downloads % head new_temp.txt
  11/1195100_24053_0_0.dragen.hard-filtered.vcf.gz
  12/1204363_24053_0_0.dragen.hard-filtered.vcf.gz
  12/1246629_24053_0_0.dragen.hard-filtered.vcf.gz
  13/1311312_24053_0_0.dragen.hard-filtered.vcf.gz
  14/1427973_24053_0_0.dragen.hard-filtered.vcf.gz
  14/1435043_24053_0_0.dragen.hard-filtered.vcf.gz
  15/1558824_24053_0_0.dragen.hard-filtered.vcf.gz
  16/1683316_24053_0_0.dragen.hard-filtered.vcf.gz
  16/1692707_24053_0_0.dragen.hard-filtered.vcf.gz
  17/1749825_24053_0_0.dragen.hard-filtered.vcf.gz
  izadia@LV19Y7325V Downloads % tail new_temp.txt
  52/5230856_24053_0_0.dragen.hard-filtered.vcf.gz
  52/5257892_24053_0_0.dragen.hard-filtered.vcf.gz
  54/5453459_24053_0_0.dragen.hard-filtered.vcf.gz
  55/5514349_24053_0_0.dragen.hard-filtered.vcf.gz
  55/5526010_24053_0_0.dragen.hard-filtered.vcf.gz
  57/5718723_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5868836_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5875882_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5889611_24053_0_0.dragen.hard-filtered.vcf.gz
  59/5917345_24053_0_0.dragen.hard-filtered.vcf.gz
  izadia@LV19Y7325V Downloads % 
  </code></pre>
  <p>After running the script I get this error</p>
  <pre><code>dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1195100_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1204363_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1246629_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1311312_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1427973_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1435043_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1558824_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1683316_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1692707_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1749825_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
  </code></pre>
  <p>because the codes starts searching from sub-folder 10 for each of files in new_temp.txt one by one</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T11:28:37.363Z
Comments.Text: So what do you want to do? If you don't want to check if the files exist, then don't check if the files exist. Are there any vcf files in your various folders that should not be processed? What is the point of `temp,txt`? And what does it contain? Full paths? FIle names?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T11:29:22.553Z
Comments.Text: Oh, and your code never checks if a file exists. Just [edit] your question, explain what you really want to do, and maybe we can help.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T11:42:01.897Z
Comments.Text: I edited my main post
Comments.UserDisplayName: Zizogolu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T11:54:32.927Z
Comments.Text: And what does `temp.txt` look like? Please show us an example. We need to know if it is one file name per line, many per line, if it is just the names or the full paths etc.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T12:45:38.500Z
Comments.Text: While you can do it in bash often switching to ie `python` gives you more debugging options, cleaner code, etc.
Comments.UserDisplayName: darked89
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T17:00:39.647Z
Comments.Text: Please do not edit your posts as follow up to answers without making the edits extremely clear to follow. Right now, we need to read the answers and your edit history (not question) side by side to understand what's going on and that's a huge pain.
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-26T17:00:46.457Z
Comments.Text: Please don't forget to mark the answer below as "accepted" if this addresses your question (it does look like it).
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-27T10:31:55.977Z
Comments.Text: Thanks for the answer but the problem have not been solved yet so the code still stats checking from sub-folder 10 for the listed files one by one which takes forever
Comments.UserDisplayName: Zizogolu
Tags: <vcf><bash><linux><wgs><for-loop>
Title: How to debug a bash script
ThreadText: # How to debug a bash script
  
  (Asked by: username_0 on 2024-02-23)
  
  Please somebody debug my code
  
  In sub-folders there are `500000 vcf files` from which I only need `500 vcf files` listed in `temp.txt` which randomly distributed in sub-folders 10 to 60. The code reads each of 500000 vcf files one by one, if a file was in temp.txt, the code runs
  
  ```
  dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
      done < "$VCF_LIST_FILE"
  
  ```
  
  on that if not, the code goes to next file such a process takes years. I want this code only locates files listed in temp.txt and run the main code on them by iteration.
  
  ```
  #!/bin/bash
  
  # Set the working directory to the desired location within DNAnexus
  
  # Number of concurrent processes
  N=4
  
  # Iterate from 10 to 60
  for folder_number in {10..60}; do
    (
      # Define the input files and parameters for this iteration
      folder_path="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/$folder_number"
      VCF_LIST_FILE="temp.txt"
      REGIONS_FILE='file.bed'
      OUTPUT_FOLDER='/out'  # Replace with your desired output folder
  
      # Set the working directory to the folder for this iteration
      dx cd "$folder_path"
  
      # Read each VCF file from the list
      while read -r vcf; do
        filename_root=$(basename "$vcf")
        output_file="${filename_root}_extracted.vcf.gz"
  
        # Use dx run swiss-army-knife to execute tabix on the VCF file
        dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
      done < "$VCF_LIST_FILE"
    ) &
  
    # Limit the number of concurrent processes
    ((i=i%N)); ((i++==0)) && wait
  done
  
  # Wait for all background jobs to finish
  wait
  
  # Finished message
  echo "++ Finished extracting genomic coordinates from VCF files"
  
  ```
  
  This is how temp.txt looks. The first two digits of vcf file names defines which sub-folder they are located. For instance, the below vcf files distributed in sub-folders 32, 33 and 34
  
  ```
  head temp_new.txt
  head: temp_new.txt: No such file or directory
  izadia@LV19Y7325V Downloads % head new_temp.txt
  11/1195100_24053_0_0.dragen.hard-filtered.vcf.gz
  12/1204363_24053_0_0.dragen.hard-filtered.vcf.gz
  12/1246629_24053_0_0.dragen.hard-filtered.vcf.gz
  13/1311312_24053_0_0.dragen.hard-filtered.vcf.gz
  14/1427973_24053_0_0.dragen.hard-filtered.vcf.gz
  14/1435043_24053_0_0.dragen.hard-filtered.vcf.gz
  15/1558824_24053_0_0.dragen.hard-filtered.vcf.gz
  16/1683316_24053_0_0.dragen.hard-filtered.vcf.gz
  16/1692707_24053_0_0.dragen.hard-filtered.vcf.gz
  17/1749825_24053_0_0.dragen.hard-filtered.vcf.gz
  izadia@LV19Y7325V Downloads % tail new_temp.txt
  52/5230856_24053_0_0.dragen.hard-filtered.vcf.gz
  52/5257892_24053_0_0.dragen.hard-filtered.vcf.gz
  54/5453459_24053_0_0.dragen.hard-filtered.vcf.gz
  55/5514349_24053_0_0.dragen.hard-filtered.vcf.gz
  55/5526010_24053_0_0.dragen.hard-filtered.vcf.gz
  57/5718723_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5868836_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5875882_24053_0_0.dragen.hard-filtered.vcf.gz
  58/5889611_24053_0_0.dragen.hard-filtered.vcf.gz
  59/5917345_24053_0_0.dragen.hard-filtered.vcf.gz
  izadia@LV19Y7325V Downloads % 
  
  ```
  
  After running the script I get this error
  
  ```
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1195100_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1204363_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1246629_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1311312_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1427973_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1435043_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1558824_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1683316_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1692707_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1749825_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
  
  ```
  
  because the codes starts searching from sub-folder 10 for each of files in new\_temp.txt one by one
  
  **Tags:** vcf, bash, linux, wgs, for-loop
  
  > **Comments:**
  > **username_1**: So what do you want to do? If you don't want to check if the files exist, then don't check if the files exist. Are there any vcf files in your various folders that should not be processed? What is the point of `temp,txt`? And what does it contain? Full paths? FIle names?  
  > **username_1**: Oh, and your code never checks if a file exists. Just [edit] your question, explain what you really want to do, and maybe we can help.  
  > **username_0**: I edited my main post  
  > **username_1**: And what does `temp.txt` look like? Please show us an example. We need to know if it is one file name per line, many per line, if it is just the names or the full paths etc.  
  > **username_2**: While you can do it in bash often switching to ie `python` gives you more debugging options, cleaner code, etc.  
  > **username_3**: Please do not edit your posts as follow up to answers without making the edits extremely clear to follow. Right now, we need to read the answers and your edit history (not question) side by side to understand what's going on and that's a huge pain.  
  > **username_4**: Please don't forget to mark the answer below as "accepted" if this addresses your question (it does look like it).  
  > **username_0**: Thanks for the answer but the problem have not been solved yet so the code still stats checking from sub-folder 10 for the listed files one by one which takes forever  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_5 on 2024-02-23. Score: 4
  
  > The first two digits of vcf file names defines which sub-folder they are located.
  
  You could use this information to edit temp.txt so it contains the full path:
  
  ```bash
  grep -v ".tbi$" temp.txt | sed 's/^\(.\)\(.\)/\1\2\/\1\2/' > new_temp.txt
  
  ```
  
  I'm assuming here that you don't actually want to run `dx run swiss-army-knife` on the vcf indexes.
  
  This will turn the example of temp.txt you gave into this:
  
  ```
  32/3295094_24053_0_0.dragen.hard-filtered.vcf.gz
  33/3309510_24053_0_0.dragen.hard-filtered.vcf.gz
  34/3468518_24053_0_0.dragen.hard-filtered.vcf.gz
  34/3471960_24053_0_0.dragen.hard-filtered.vcf.gz
  
  ```
  
  You can then simply run a single for loop over that, instead of the nested one you are using now.
  
  eg.
  
  ```bash
  while read vcf
    do
      dx run swiss-army-knife [...]
    done <  new_temp.txt
  
  ```
  
  
  ---
  
  