Answers.Body: <p>Proximity to NCBI may not necessarily give you the fastest transfer speed. AWS may be deliberately throttling the Internet connection to limit the likelihood that people will use it for undesirable things. There's a chance that a home network might be faster, but you're likely to get the fastest connection to NCBI by using an academic system that is linked to NCBI via a research network.</p>
  
  <p>Another possibility is using Aspera for downloads. This is unlikely to help if bandwidth is being throttled, but it might help if there's a bit of congestion through the regular methods:</p>
  
  <p><a href="https://www.ncbi.nlm.nih.gov/public/" rel="noreferrer">https://www.ncbi.nlm.nih.gov/public/</a></p>
  
  <p>NCBI also has an online <a href="https://www.ncbi.nlm.nih.gov/books/NBK51062/?report=reader" rel="noreferrer">book</a> about best practises for downloading data from their servers.</p>
  
  <p>On a related note, just in case someone sees this and EBI/ENA is an option, there's a great guide for how to do file transfer using Aspera on the EBI web site:</p>
  
  <p><a href="https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera" rel="noreferrer">https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera</a></p>
  
  <blockquote>
    <p>Your command should look similar to this on Unix:</p>
  </blockquote>
  
  <pre><code>ascp -QT -l 300m -i &lt;aspera connect installation directory&gt;/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:&lt;file or files to download&gt; &lt;download location&gt;
  </code></pre>
  
  <p>In my case, I've just started downloading some files from a MinION sequencing run. The estimated completion time via standard FTP was 12 hours for about 32GB of data; <code>ascp</code> has reduced that estimated download time to about an hour. Here's the command I used for downloading:</p>
  
  <pre><code>ascp -QT -l 300m -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/ERA932/ERA932268/oxfordnanopore_native/20160804_Mock.tar.gz .
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-03T12:10:48.387Z
Answers.LastActivityDate: 2017-06-05T02:20:43.573Z
Answers.LastEditDate: 2017-06-05T02:20:43.573Z
Answers.UserDisplayName: gringer
Answers.Body: <p>By far the fastest method in my experience has been to use the <a href="http://bioconductor.org/packages/release/bioc/html/SRAdb.html" rel="nofollow noreferrer">SRAdb</a> library in R. For most entries, you can download fastq files directly. Some older experiments don't have them, but I've still found it much faster to download SRA files via <code>getSRAfile()</code> and then to convert them using <code>fastqdump</code> than to use <code>fastqdump</code> directly. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-06T23:14:28.103Z
Answers.Text: +1 for providing an R solution. That said, I don't think NCBI keeps FASTQ. I speculate that SRAdb just downloads from ENA. [Its source code](https://github.com/Bioconductor-mirror/SRAdb/blob/release-3.5/R/getFASTQfile.R) seems to confirm my guess.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-06T07:56:48.763Z
Answers.LastActivityDate: 2017-06-06T07:56:48.763Z
Answers.UserDisplayName: Sarah Carl
Answers.Body: <p>In general, the best way to download SRA data is: don't download from SRA. However, as ENA has not be sync'd yet, I would recommend to download from SRA ftp and then convert to fastq locally. You can find files in the SRA format <a href="https://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/" rel="noreferrer">here</a>. Downloading and then converting locally is much faster than direct retrieval from NCBI for some mysterious reasons.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-13T09:00:22.667Z
Answers.Text: Agree. I always download via ENA, it's reliably fast and you get the fastq files natively. I usually search with SRA as it's terrible on ENA.
Answers.UserDisplayName: ithinkiam
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-06T02:47:36.653Z
Answers.LastActivityDate: 2017-06-06T02:47:36.653Z
Answers.UserDisplayName: user172818
Answers.Body: <p>NCBI is moving slowly to cloud (AWS/Google) for hosting. As such the tradiational FTP urls do not always work. My solution involves using <a href="https://github.com/saketkc/pysradb" rel="nofollow noreferrer">pysradb</a>.</p>
  
  <p>An entire project can be downloaded by:</p>
  
  <pre><code>pysradb download -p &lt;SRP_ID&gt;
  </code></pre>
  
  <p>A Google Colab notebook with metadata retrieval commands is <a href="https://colab.research.google.com/drive/1C60V-jkcNZiaCra_V5iEyFs318jgVoUR" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-25T15:30:33.290Z
Answers.Text: Thanks thats good to know.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T00:21:28.930Z
Answers.LastActivityDate: 2020-02-06T00:21:28.930Z
Answers.UserDisplayName: Saket Choudhary
Answers.Body: <p>I recommend using <a href="https://github.com/rpetit3/fastq-dl" rel="nofollow noreferrer">fastq-dl</a>: downloads from either ENA or SRA, multithreaded, automatic retries.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T10:26:47.190Z
Answers.LastActivityDate: 2023-08-03T10:26:47.190Z
Answers.UserDisplayName: bricoletc
Body: <p>I'm trying to download three WGS datasets from the SRA that are each between 60 and 100GB in size.  So far I've tried:</p>
  
  <ul>
  <li>Fetching the .sra files directly from NCBI's ftp site</li>
  <li>Fetching the .sra files directly using the aspera command line (<code>ascp</code>)</li>
  <li>Using the SRA toolkit's <code>fastqdump</code> and <code>samdump</code> tools</li>
  </ul>
  
  <p>It's excruciatingly slow.  I've had three <code>fastqdump</code> processes running in parallel now for approximately 18 hours.  They're running on a large AWS instance in the US east (Virginia) region, which I figure is about as close to NCBI as I can get.  In 18 hours they've downloaded a total of 33GB of data.  By my calculation that's ~500kb/s.  They do appear to still be running - the fastq files continue to grow and their timestamps continue to update.</p>
  
  <p>At this rate it's going to take me days or weeks just to download the datasets.  Surely the SRA must be capable of moving data at higher rates that this?  I've also looked, and unfortunately the datasets I'm interested have not been mirrored out to ENA or the Japanese archive, so it looks like I'm stuck working with the SRA.</p>
  
  <p>Is there a better way to fetch this data that wouldn't take multiple days?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-03T12:39:47.973Z
Comments.Text: The SRA is definitely capable of moving this data at higher rates. The throttling happens not at their end but somewhere else in the network connection. In my test just now, using `fastq-dump`, I get a throughput of ~11.5 MiB/s, more than 20 times the rate you observe.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-05T08:55:36.787Z
Comments.Text: I don't know if this can help answering your question, but providing the commands you used might be at least useful for other readers.
Comments.UserDisplayName: bli
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-06T09:44:57.877Z
Comments.Text: In general I find that its the conversion from SRA to fastq that is slow, not the downloading. You can try this by running NCBI's prefetch tool first to download the file (through using ascp if possible) and then `fastq-dump`ing it. Also note that ascp works better when you limit the band width its allowed. We limit it to 100m/sec.
Comments.UserDisplayName: Ian Sudbery
Tags: <data-download>
Title: What's the best way to download data from the SRA? Is it really this slow?
ThreadText: # What's the best way to download data from the SRA? Is it really this slow?
  
  (Asked by: username_0 on 2017-06-03)
  
  I'm trying to download three WGS datasets from the SRA that are each between 60 and 100GB in size. So far I've tried:
  
  * Fetching the .sra files directly from NCBI's ftp site
  * Fetching the .sra files directly using the aspera command line (`ascp`)
  * Using the SRA toolkit's `fastqdump` and `samdump` tools
  
  It's excruciatingly slow. I've had three `fastqdump` processes running in parallel now for approximately 18 hours. They're running on a large AWS instance in the US east (Virginia) region, which I figure is about as close to NCBI as I can get. In 18 hours they've downloaded a total of 33GB of data. By my calculation that's ~500kb/s. They do appear to still be running - the fastq files continue to grow and their timestamps continue to update.
  
  At this rate it's going to take me days or weeks just to download the datasets. Surely the SRA must be capable of moving data at higher rates that this? I've also looked, and unfortunately the datasets I'm interested have not been mirrored out to ENA or the Japanese archive, so it looks like I'm stuck working with the SRA.
  
  Is there a better way to fetch this data that wouldn't take multiple days?
  
  **Tags:** data-download
  
  > **Comments:**
  > **username_1**: The SRA is definitely capable of moving this data at higher rates. The throttling happens not at their end but somewhere else in the network connection. In my test just now, using `fastq-dump`, I get a throughput of ~11.5 MiB/s, more than 20 times the rate you observe.  
  > **username_2**: I don't know if this can help answering your question, but providing the commands you used might be at least useful for other readers.  
  > **username_3**: In general I find that its the conversion from SRA to fastq that is slow, not the downloading. You can try this by running NCBI's prefetch tool first to download the file (through using ascp if possible) and then `fastq-dump`ing it. Also note that ascp works better when you limit the band width its allowed. We limit it to 100m/sec.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_4 on 2017-06-03. Score: 12
  
  Proximity to NCBI may not necessarily give you the fastest transfer speed. AWS may be deliberately throttling the Internet connection to limit the likelihood that people will use it for undesirable things. There's a chance that a home network might be faster, but you're likely to get the fastest connection to NCBI by using an academic system that is linked to NCBI via a research network.
  
  Another possibility is using Aspera for downloads. This is unlikely to help if bandwidth is being throttled, but it might help if there's a bit of congestion through the regular methods:
  
  <https://www.ncbi.nlm.nih.gov/public/>
  
  NCBI also has an online [book](https://www.ncbi.nlm.nih.gov/books/NBK51062/?report=reader) about best practises for downloading data from their servers.
  
  On a related note, just in case someone sees this and EBI/ENA is an option, there's a great guide for how to do file transfer using Aspera on the EBI web site:
  
  <https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera>
  
  > Your command should look similar to this on Unix:
  
  ```
  ascp -QT -l 300m -i <aspera connect installation directory>/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:<file or files to download> <download location>
  
  ```
  
  In my case, I've just started downloading some files from a MinION sequencing run. The estimated completion time via standard FTP was 12 hours for about 32GB of data; `ascp` has reduced that estimated download time to about an hour. Here's the command I used for downloading:
  
  ```
  ascp -QT -l 300m -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/ERA932/ERA932268/oxfordnanopore_native/20160804_Mock.tar.gz .
  
  ```
  
  
  ---
  
  ### Answer by username_6 on 2017-06-06. Score: 6
  
  In general, the best way to download SRA data is: don't download from SRA. However, as ENA has not be sync'd yet, I would recommend to download from SRA ftp and then convert to fastq locally. You can find files in the SRA format [here](https://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/). Downloading and then converting locally is much faster than direct retrieval from NCBI for some mysterious reasons.
  
  > **Comments:**
  > **username_7**: Agree. I always download via ENA, it's reliably fast and you get the fastq files natively. I usually search with SRA as it's terrible on ENA.  
  
  ---
  
  ### Answer by username_5 on 2017-06-06. Score: 3
  
  By far the fastest method in my experience has been to use the [SRAdb](http://bioconductor.org/packages/release/bioc/html/SRAdb.html) library in R. For most entries, you can download fastq files directly. Some older experiments don't have them, but I've still found it much faster to download SRA files via `getSRAfile()` and then to convert them using `fastqdump` than to use `fastqdump` directly.
  
  > **Comments:**
  > **username_6**: +1 for providing an R solution. That said, I don't think NCBI keeps FASTQ. I speculate that SRAdb just downloads from ENA. [Its source code](https://github.com/Bioconductor-mirror/SRAdb/blob/release-3.5/R/getFASTQfile.R) seems to confirm my guess.  
  
  ---
  
  ### Answer by username_8 on 2020-02-06. Score: 2
  
  NCBI is moving slowly to cloud (AWS/Google) for hosting. As such the tradiational FTP urls do not always work. My solution involves using [pysradb](https://github.com/saketkc/pysradb).
  
  An entire project can be downloaded by:
  
  ```
  pysradb download -p <SRP_ID>
  
  ```
  
  A Google Colab notebook with metadata retrieval commands is [here](https://colab.research.google.com/drive/1C60V-jkcNZiaCra_V5iEyFs318jgVoUR).
  
  > **Comments:**
  > **username_9**: Thanks thats good to know.  
  
  ---
  
  ### Answer by username_10 on 2023-08-03. Score: 1
  
  I recommend using [fastq-dl](https://github.com/rpetit3/fastq-dl): downloads from either ENA or SRA, multithreaded, automatic retries.
  
  
  ---
  
  