Answers.Body: <p>I once needed to download more than 1000 files from SRA, and I found a blog post that is very useful (<a href="https://reneshbedre.github.io/blog/fqutil.html" rel="nofollow noreferrer">https://reneshbedre.github.io/blog/fqutil.html</a>). The idea is to use aspera to fetch SRA files. This is fast becasue of aspera. Then use fasterq-dump to convert SRA file to fastq. fasterq-dump is faster than fastq-dump partly because it does not compress the output file. I think compressing is the rate-limiting step here. After this, you can use pigz, which is a multi-threading version of gzip to compress the fastq files. Note that if you want to download files to a computing cluster, you can submit jobs for fasterq-dump and pigz. This makes things much faster. You cannot submit jobs for fetching SRA, because you need to connect to internet.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T15:59:35.270Z
Answers.Text: Thanks for sharing this. So I tried running prefetch with aspera as explained in the blog, but I get unrecognized option when I include the '-a' option. I do have aspera installed. Did you have this issue by any chance?
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T16:32:43.377Z
Answers.Text: seems like aspera is no longer supported https://github.com/ncbi/sra-tools/issues/255
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T16:33:18.447Z
Answers.Text: however, prefetch is still very fast, thanks for sharing this, update your answer to reflect this and I'll accept it!
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T15:07:51.370Z
Answers.LastActivityDate: 2020-07-13T15:07:51.370Z
Answers.UserDisplayName: Phoenix Mu
Body: <p>I have the following code to download data from the SRA using multithreading in Python. After running this a few times now (for testing purposes), I keep getting denied access to the data. Not sure how to fix this. In particular, <code>cat</code>ing the output files gives:</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;&lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access Denied&lt;/Message&gt;&lt;RequestId&gt;cb0f0e98-cafb-1dd7-9b7b-d8c49756ec52&lt;/RequestId&gt;&lt;HostId&gt;QeDGVwBXYp61J0B4_OUTn7UsEsiQEec0n18DAeR0kaE&lt;/HostId&gt;&lt;/Error&gt;
  </code></pre>
  <p>Here is my code:</p>
  <pre><code>import threading
  import requests
  import time 
  
  start = time.perf_counter()
  
  class MyThread(threading.Thread):
      def __init__(self, url):
          threading.Thread.__init__(self)
          self.url = url
          self.result = None
          self.filename = url.split('/')[-1]
      def run(self):
          res = requests.get(self.url)
          with open(self.filename, 'wb') as f:
              f.write(res.content)
  urls = [
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.1',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.2',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.1',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.2']
  
  threads = [MyThread(url, ) for url in urls]
  for thread in threads:
      thread.start()
  for thread in threads:
      thread.join()
  
  finish = time.perf_counter()
  
  print(f'Finished in {round(finish-start, 2)} second(s)')
  </code></pre>
  
Tags: <python>
Title: What is the best way to get a large number of RNA seq data from SRA in Python without being denied access
ThreadText: # What is the best way to get a large number of RNA seq data from SRA in Python without being denied access
  
  (Asked by: username_0 on 2020-07-12)
  
  I have the following code to download data from the SRA using multithreading in Python. After running this a few times now (for testing purposes), I keep getting denied access to the data. Not sure how to fix this. In particular, `cat`ing the output files gives:
  
  ```
  <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>cb0f0e98-cafb-1dd7-9b7b-d8c49756ec52</RequestId><HostId>QeDGVwBXYp61J0B4_OUTn7UsEsiQEec0n18DAeR0kaE</HostId></Error>
  
  ```
  
  Here is my code:
  
  ```
  import threading
  import requests
  import time 
  
  start = time.perf_counter()
  
  class MyThread(threading.Thread):
      def __init__(self, url):
          threading.Thread.__init__(self)
          self.url = url
          self.result = None
          self.filename = url.split('/')[-1]
      def run(self):
          res = requests.get(self.url)
          with open(self.filename, 'wb') as f:
              f.write(res.content)
  urls = [
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.1',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.2',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.1',
          'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.2']
  
  threads = [MyThread(url, ) for url in urls]
  for thread in threads:
      thread.start()
  for thread in threads:
      thread.join()
  
  finish = time.perf_counter()
  
  print(f'Finished in {round(finish-start, 2)} second(s)')
  
  ```
  
  **Tags:** python
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-13. Score: 0 | Accepted answer
  
  I once needed to download more than 1000 files from SRA, and I found a blog post that is very useful (<https://reneshbedre.github.io/blog/fqutil.html>). The idea is to use aspera to fetch SRA files. This is fast becasue of aspera. Then use fasterq-dump to convert SRA file to fastq. fasterq-dump is faster than fastq-dump partly because it does not compress the output file. I think compressing is the rate-limiting step here. After this, you can use pigz, which is a multi-threading version of gzip to compress the fastq files. Note that if you want to download files to a computing cluster, you can submit jobs for fasterq-dump and pigz. This makes things much faster. You cannot submit jobs for fetching SRA, because you need to connect to internet.
  
  > **Comments:**
  > **username_0**: Thanks for sharing this. So I tried running prefetch with aspera as explained in the blog, but I get unrecognized option when I include the '-a' option. I do have aspera installed. Did you have this issue by any chance?  
  > **username_0**: seems like aspera is no longer supported https://github.com/ncbi/sra-tools/issues/255  
  > **username_0**: however, prefetch is still very fast, thanks for sharing this, update your answer to reflect this and I'll accept it!  
  
  ---
  
  