Answers.Body: <p>The main problem is that your script is inadvertently treating a set of file and directory names as a single SRA accession number, all in one variable.  You should update it to work with each accession separately and make sure they really are accessions.</p>
  <hr />
  <p>On this line you're taking all the names of things in your working directory (minus the script), each on a separate line, and trimming .sra off the end of each line:</p>
  <pre><code>sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')
  </code></pre>
  <p>In your case that list was apparently just a directory named &quot;sra&quot;.  (Even though I don't have that directory on my end, you can see it in your output, from the first of two <code>ls *</code> at the end, where you have <code>sra/SRR12332899.sra</code>.)  So later on it tries to <code>fastq-dump sra</code>, which gives you that &quot;cannot be opened as database or table&quot; error.  (<code>vdb-validate</code> can take a directory as input, so it doesn't complain.)  If I run your script from scratch in an empty directory with SRR12332899, it actually seems to run OK, aside from removing everything it creates at the end.  But if you run it with a second accession you see the problem, since it then tries things like <code>vdb-validate &quot;several accessions and filenames separated by newlines&quot;</code>.</p>
  <p>I think a proper fix would involve a bunch of changes, but here are some tips, at least:</p>
  <ul>
  <li>use subdirectories to keep things organized so you don't have to juggle multiple types of file in one place</li>
  <li><code>set -x</code> at the start of a script will show you each command as it happens, which can be very helpful for troubleshooting</li>
  <li>be careful with quotes and whitespace in bash.  If you want to loop over each word in a variable you can do <code>for thing in $var; do ...; done</code> (unquoted) but if you run into filenames with whitespace, it breaks them up.  Another pattern is <code>command | while read thing; do ...; done</code> which will go line-by-line.  (In that case if you were working with a variable storing the command output, you might do something like <code>echo &quot;$output&quot; | while...</code>, quoted that time, since you'd want to preserve the newlines.)</li>
  <li><code>ls *</code> will expand each file/directory name as separate arguments (so bash really ends up calling <code>ls &quot;sra&quot; ...whatever else...</code>), and when ls sees directory names is lists their contents too.  If you just want to show all the files/directories you can leave off the asterisk altogether, or if you wanted to give directory names but not have it list their contents, you can use <code>ls -d</code>.</li>
  <li>you don't necessarily have to call <code>ls</code> to store the names as a block of text.  You can loop over each filename, including those with spaces, directly in bash (e.g. <code>for thing in *; do echo &quot;path: '$thing'&quot;; done</code>)</li>
  <li>be careful with the rm command, especially combined with wildcards, and especially with <code>-rf</code>.  If <code>sraID</code> happened to be empty somehow, think of what <code>rm -rf &quot;$sraID&quot;*</code> would actually mean.  (And yes, scripts can remove themselves!)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T16:31:57.907Z
Answers.Text: will work on your suggestion and show you what i have done
Answers.UserDisplayName: PesKchan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-05T21:40:09.380Z
Answers.LastActivityDate: 2023-08-05T21:40:09.380Z
Answers.UserDisplayName: Jesse
Body: <p>I have a very basic objective which I want to give list of id to a ncbi-srafastq tool kit which will prefetch the id which is in .sra file extension and then it will run fastq-dump on the file to split the files into fastq files, single file if there is single end file or paired end file if its paired end or two set of files.</p>
  <p>This is my code</p>
  <pre><code>#!/bin/bash
  
  input=$1
  
  a=<span class="math-container">$(basename "$</span>input&quot;)
  id=${a/.id/}
  
  # Download SRA data
  prefetch &quot;$id&quot;
  
  # Extract SRA ID from the downloaded data
  #sraID=<span class="math-container">$(ls -1 | grep -v run.sh)
  sraID=$</span>(ls -1 | grep -v run.sh | sed 's/\.sra<span class="math-container">$//')
  # Validate the downloaded SRA data
  vdb-validate "$</span>sraID&quot;
  
  # Dump fastq data from the SRA file
  fastq-dump &quot;$sraID&quot;  --gzip
  
  # Concatenate fastq files for read1, read2, and unpaired reads (if available)
  for s in &quot;<span class="math-container">$sraID"
  do
    echo "$</span>s&quot;
    if [ &quot;<span class="math-container">$s" != "$</span>id&quot; ]
    then
      [ -f &quot;<span class="math-container">${s}_1.fastq" ] &amp;&amp; cat "$</span>{s}_1.fastq&quot; &gt;&gt; &quot;<span class="math-container">${id}_R1.fastq"
      [ -f "$</span>{s}_2.fastq&quot; ] &amp;&amp; cat &quot;<span class="math-container">${s}_2.fastq" &gt;&gt; "$</span>{id}_R2.fastq&quot;
      [ -f &quot;<span class="math-container">${s}.fastq" ] &amp;&amp; cat "$</span>{s}.fastq&quot; &gt;&gt; &quot;${id}.fastq&quot;
    fi
  done
  
  # Compress fastq files using pigz
  #for i in &quot;<span class="math-container">${id}"*.fastq
  #do
    #echo "$</span>i&quot;
   # pigz &quot;$i&quot; &amp; sleep 1
  #done
  #wait
  
  # Remove the original uncompressed fastq file if both read1 and read2 files are present
  [ -f &quot;<span class="math-container">${id}_R1.fastq.gz" ] &amp;&amp; [ -f "$</span>{id}_R2.fastq.gz&quot; ] &amp;&amp; [ -f &quot;<span class="math-container">${id}.fastq.gz" ] &amp;&amp; rm "$</span>{id}.fastq.gz&quot;
  
  # Test deleting intermediate folder
  echo &quot;Space before tmp folder delete&quot;
  echo &quot;Removing tmp folder <span class="math-container">$sraID "
  ls *
  rm -rf "$</span>sraID&quot;*
  ls *
  echo 'done'
  </code></pre>
  <p>I run using as small example file as such</p>
  <pre><code>./run.sh SRR12332899
  </code></pre>
  <p>The error I get is this</p>
  <pre><code>2023-08-05T12:37:32 prefetch.3.0.6: Current preference is set to retrieve SRA Normalized Format files with full base quality scores.
  2023-08-05T12:37:33 prefetch.3.0.6: 1) Downloading 'SRR12332899'...
  2023-08-05T12:37:33 prefetch.3.0.6: SRA Normalized Format file is being retrieved, if this is different from your preference, it may be due to current file availability.
  2023-08-05T12:37:33 prefetch.3.0.6:  Downloading via HTTPS...
  2023-08-05T12:40:48 prefetch.3.0.6:  HTTPS download succeed
  2023-08-05T12:40:48 prefetch.3.0.6:  'SRR12332899' is valid
  2023-08-05T12:40:48 prefetch.3.0.6: 1) 'SRR12332899' was downloaded successfully
  2023-08-05T12:40:48 prefetch.3.0.6: 'SRR12332899' has 0 unresolved dependencies
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' metadata: md5 ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Table 'SEQUENCE' metadata: md5 ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'ALTREAD': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'QUALITY': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'READ': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'SPOT_GROUP': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'sra/SRR12332899.sra' contains only unaligned reads
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' is consistent
  2023-08-05T12:40:49 fastq-dump.3.0.6 err: item not found while constructing within virtual database module - the path 'sra' cannot be opened as database or table
  fastq-dump quit with error code 3
  sra
  Space before tmp folder delete
  Removing tmp folder sra
  run.sh
  
  sra:
  SRR12332899.sra
  run.sh
  done
  </code></pre>
  <p>If I do just</p>
  <p><code>prefetch SRR12332899</code> and then <code>fastq-dump SRR12332899</code></p>
  <p>I get a SRR12332899.fastq  SRR12332899.sra</p>
  <p>This is for one file since I want to download list of files using the script which is not working.</p>
  <p>Any suggestion would be really helpful</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-07T11:09:54.333Z
Comments.Text: Note that you are catting an already gzipped files since you used --gzip in fastq-dump, so the suffix of the output must be .gz
  
  It would be muchfaster to simply search for the dataset at sra-explorer.info and get fastq download links right away.
Comments.UserDisplayName: ATpoint
Tags: <fastq><shell><ncbi>
Title: ncbi fastq dump error in loop
ThreadText: # ncbi fastq dump error in loop
  
  (Asked by: username_0 on 2023-08-05)
  
  I have a very basic objective which I want to give list of id to a ncbi-srafastq tool kit which will prefetch the id which is in .sra file extension and then it will run fastq-dump on the file to split the files into fastq files, single file if there is single end file or paired end file if its paired end or two set of files.
  
  This is my code
  
  ```
  #!/bin/bash
  
  input=$1
  
  a=$(basename "$input")
  id=${a/.id/}
  
  # Download SRA data
  prefetch "$id"
  
  # Extract SRA ID from the downloaded data
  #sraID=$(ls -1 | grep -v run.sh)
  sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')
  # Validate the downloaded SRA data
  vdb-validate "$sraID"
  
  # Dump fastq data from the SRA file
  fastq-dump "$sraID"  --gzip
  
  # Concatenate fastq files for read1, read2, and unpaired reads (if available)
  for s in "$sraID"
  do
    echo "$s"
    if [ "$s" != "$id" ]
    then
      [ -f "${s}_1.fastq" ] && cat "${s}_1.fastq" >> "${id}_R1.fastq"
      [ -f "${s}_2.fastq" ] && cat "${s}_2.fastq" >> "${id}_R2.fastq"
      [ -f "${s}.fastq" ] && cat "${s}.fastq" >> "${id}.fastq"
    fi
  done
  
  # Compress fastq files using pigz
  #for i in "${id}"*.fastq
  #do
    #echo "$i"
   # pigz "$i" & sleep 1
  #done
  #wait
  
  # Remove the original uncompressed fastq file if both read1 and read2 files are present
  [ -f "${id}_R1.fastq.gz" ] && [ -f "${id}_R2.fastq.gz" ] && [ -f "${id}.fastq.gz" ] && rm "${id}.fastq.gz"
  
  # Test deleting intermediate folder
  echo "Space before tmp folder delete"
  echo "Removing tmp folder $sraID "
  ls *
  rm -rf "$sraID"*
  ls *
  echo 'done'
  
  ```
  
  I run using as small example file as such
  
  ```
  ./run.sh SRR12332899
  
  ```
  
  The error I get is this
  
  ```
  2023-08-05T12:37:32 prefetch.3.0.6: Current preference is set to retrieve SRA Normalized Format files with full base quality scores.
  2023-08-05T12:37:33 prefetch.3.0.6: 1) Downloading 'SRR12332899'...
  2023-08-05T12:37:33 prefetch.3.0.6: SRA Normalized Format file is being retrieved, if this is different from your preference, it may be due to current file availability.
  2023-08-05T12:37:33 prefetch.3.0.6:  Downloading via HTTPS...
  2023-08-05T12:40:48 prefetch.3.0.6:  HTTPS download succeed
  2023-08-05T12:40:48 prefetch.3.0.6:  'SRR12332899' is valid
  2023-08-05T12:40:48 prefetch.3.0.6: 1) 'SRR12332899' was downloaded successfully
  2023-08-05T12:40:48 prefetch.3.0.6: 'SRR12332899' has 0 unresolved dependencies
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' metadata: md5 ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Table 'SEQUENCE' metadata: md5 ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'ALTREAD': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'QUALITY': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'READ': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'SPOT_GROUP': checksums ok
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'sra/SRR12332899.sra' contains only unaligned reads
  2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' is consistent
  2023-08-05T12:40:49 fastq-dump.3.0.6 err: item not found while constructing within virtual database module - the path 'sra' cannot be opened as database or table
  fastq-dump quit with error code 3
  sra
  Space before tmp folder delete
  Removing tmp folder sra
  run.sh
  
  sra:
  SRR12332899.sra
  run.sh
  done
  
  ```
  
  If I do just
  
  `prefetch SRR12332899` and then `fastq-dump SRR12332899`
  
  I get a SRR12332899.fastq SRR12332899.sra
  
  This is for one file since I want to download list of files using the script which is not working.
  
  Any suggestion would be really helpful
  
  **Tags:** fastq, shell, ncbi
  
  > **Comments:**
  > **username_1**: Note that you are catting an already gzipped files since you used --gzip in fastq-dump, so the suffix of the output must be .gz  It would be muchfaster to simply search for the dataset at sra-explorer.info and get fastq download links right away.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-05. Score: 3 | Accepted answer
  
  The main problem is that your script is inadvertently treating a set of file and directory names as a single SRA accession number, all in one variable. You should update it to work with each accession separately and make sure they really are accessions.
  
  ---
  
  On this line you're taking all the names of things in your working directory (minus the script), each on a separate line, and trimming .sra off the end of each line:
  
  ```
  sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')
  
  ```
  
  In your case that list was apparently just a directory named "sra". (Even though I don't have that directory on my end, you can see it in your output, from the first of two `ls *` at the end, where you have `sra/SRR12332899.sra`.) So later on it tries to `fastq-dump sra`, which gives you that "cannot be opened as database or table" error. (`vdb-validate` can take a directory as input, so it doesn't complain.) If I run your script from scratch in an empty directory with SRR12332899, it actually seems to run OK, aside from removing everything it creates at the end. But if you run it with a second accession you see the problem, since it then tries things like `vdb-validate "several accessions and filenames separated by newlines"`.
  
  I think a proper fix would involve a bunch of changes, but here are some tips, at least:
  
  * use subdirectories to keep things organized so you don't have to juggle multiple types of file in one place
  * `set -x` at the start of a script will show you each command as it happens, which can be very helpful for troubleshooting
  * be careful with quotes and whitespace in bash. If you want to loop over each word in a variable you can do `for thing in $var; do ...; done` (unquoted) but if you run into filenames with whitespace, it breaks them up. Another pattern is `command | while read thing; do ...; done` which will go line-by-line. (In that case if you were working with a variable storing the command output, you might do something like `echo "$output" | while...`, quoted that time, since you'd want to preserve the newlines.)
  * `ls *` will expand each file/directory name as separate arguments (so bash really ends up calling `ls "sra" ...whatever else...`), and when ls sees directory names is lists their contents too. If you just want to show all the files/directories you can leave off the asterisk altogether, or if you wanted to give directory names but not have it list their contents, you can use `ls -d`.
  * you don't necessarily have to call `ls` to store the names as a block of text. You can loop over each filename, including those with spaces, directly in bash (e.g. `for thing in *; do echo "path: '$thing'"; done`)
  * be careful with the rm command, especially combined with wildcards, and especially with `-rf`. If `sraID` happened to be empty somehow, think of what `rm -rf "$sraID"*` would actually mean. (And yes, scripts can remove themselves!)
  
  > **Comments:**
  > **username_0**: will work on your suggestion and show you what i have done  
  
  ---
  
  