Answers.Body: <p>Answers to this question can be found on <a href="https://www.biostars.org/p/91885/" rel="noreferrer">biostars</a>. To give you a short summary:</p>
  
  <ol>
  <li><p>Faster way might be to use the <code>parallel-fastq-dump</code>, as suggested in this <a href="https://www.biostars.org/p/91885/#300457" rel="noreferrer">answer</a>. I never tested that tool though, my own experience is that <code>prefetch</code> is more stable than <code>fastq-dump</code> command.</p></li>
  <li><p>In that same post it seems that the Japanese mirror seems faster, don't know if that is still valid.</p></li>
  <li><p>The reason this is never optimized might be that downloading from <code>SRA</code> is not a task you'll have to do repeatedly, as described here in this <a href="https://www.biostars.org/p/91885/#91895" rel="noreferrer">post</a>.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T10:02:46.080Z
Answers.Text: Thanks, comprehensive answer. Is it this site https://www.ddbj.nig.ac.jp/  ?
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T10:17:37.637Z
Answers.Text: https://www.ddbj.nig.ac.jp/dra/index-e.html, yes it is the same.
Answers.UserDisplayName: benn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T08:36:07.600Z
Answers.LastActivityDate: 2019-04-24T08:36:07.600Z
Answers.UserDisplayName: benn
Answers.Body: <p>You can try <code>wget</code> to download SRA files from NCBI server. This seems to be 2x faster in terms of CPU time than <code>prefetch</code> and has the advantage of not clogging your <code>/home/ncbi/public/sra/</code> directory in case you have space limitation. The tutorial for <code>wget</code> downloading method is available at <a href="https://github.com/johanzi/ncbi_tutorial" rel="nofollow noreferrer">https://github.com/johanzi/ncbi_tutorial</a>.</p>
  
  <p>Here my timing of the <code>prefetch</code> method:</p>
  
  <pre><code>time prefetch SRR3300113
  real    3m5.609s
  user    0m6.332s
  sys     0m2.076s
  </code></pre>
  
  <p>Versus the <code>wget</code> method:</p>
  
  <pre><code>time wget ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR330/SRR3300113/SRR3300113.sra
  real    3m4.696s
  user    0m0.456s
  sys     0m3.332s
  </code></pre>
  
  <p>That makes a user+sys time of 8.402s for <code>prefetch</code> and 3.788s for <code>wget</code>. Not so much difference for real time though (about a second).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-29T15:56:38.050Z
Answers.Text: Hello Johan, welcome to BIoinformatics.SE. Is there a reason to think that `wget` would be faster? Intuitively I would guess that it's the server that is clogged, then using a different program to query the server does not sound like something that would help. (by the way, my inability to see how it could help does not mean that it does not :-)
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-30T14:55:09.850Z
Answers.Text: Hi Kamil, the `wget` process is about 2x faster in terms of CPU time than `prefetch` method based on my test:
Answers.UserDisplayName: Johan Zicola
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T07:47:41.540Z
Answers.Text: `user` + `sys` sort of tells you how long CPUs were active during the `real` time of execution (i.e. for user `real` is what matters, you can check [this](https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1)). So, what I would get out of your nice test is, that the `wget` is a more optimized tool, but both of them are fetching the data from exactly the same place and the limiting factor is the speed of their ftp.
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T11:52:46.773Z
Answers.Text: Yes. I would say the main advantage of `wget` is that you can choose where to put your downloaded SRA files while it goes in default into your home directory `/home/user/ncbi/public/sra` with `prefetch`. It can be a problem when you work on a server and you have limited disk usage for your home directory.
Answers.UserDisplayName: Johan Zicola
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-20T02:53:44.593Z
Answers.Text: You can choose where to put the cache, at least as of a few versions ago, via vdb-config. I agree it's a total pain that it defaults to the home directory instead of the current one, but it's a fixable problem.
Answers.UserDisplayName: GenesRus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-29T11:59:58.650Z
Answers.LastActivityDate: 2019-05-01T14:34:27.090Z
Answers.LastEditDate: 2019-05-01T14:34:27.090Z
Answers.UserDisplayName: Johan Zicola
Body: <p>I have 2 .csv files that contains a list of accession codes. For example for this <a href="https://www.ncbi.nlm.nih.gov/biosample?LinkName=bioproject_biosample_all&amp;from_uid=520914" rel="nofollow noreferrer">experiment</a> a .csv file will contain:</p>
  <pre><code>SAMN10870244
  SAMN10870243
  SAMN10870242
  SAMN10870241
  SAMN10870240
  SAMN10870239
  SAMN10870238
  SAMN10870237
  SAMN10870236
  SAMN10870235
  SAMN10870234
  SAMN10870233
  </code></pre>
  <p>Then I run the following</p>
  <pre><code>for i in <span class="math-container">$(cat *.csv); do echo $</span>{i}; /usr/bin/sratoolkit/bin/fastq-dump --gzip ${i} &amp; done
  </code></pre>
  <ol>
  <li>Some of my downloads takes a lot of time days. Is there a faster way to get stuff from NCBI?</li>
  <li>Are there mirrors of NCBI to accelerate the process?</li>
  <li>Why is it so slow?</li>
  </ol>
  <hr />
  <p><strong>EDIT 2024</strong><br />
  You might want to look into:</p>
  <ul>
  <li><a href="https://github.com/kblin/ncbi-acc-download" rel="nofollow noreferrer"><code>ncbi-acc-download</code></a>: A CLI tool for efficient downloads from NCBI Entrez.</li>
  <li><a href="https://nsdpy.readthedocs.io/en/latest/main.html" rel="nofollow noreferrer"><code>NSDPY</code></a>: A Python library for batch downloading with additional features like taxonomic sorting.</li>
  </ul>
  
Tags: <phylogenetics>
Title: How do you download stuff from NCBI fast?
ThreadText: # How do you download stuff from NCBI fast?
  
  (Asked by: username_0 on 2019-04-24)
  
  I have 2 .csv files that contains a list of accession codes. For example for this [experiment](https://www.ncbi.nlm.nih.gov/biosample?LinkName=bioproject_biosample_all&from_uid=520914) a .csv file will contain:
  
  ```
  SAMN10870244
  SAMN10870243
  SAMN10870242
  SAMN10870241
  SAMN10870240
  SAMN10870239
  SAMN10870238
  SAMN10870237
  SAMN10870236
  SAMN10870235
  SAMN10870234
  SAMN10870233
  
  ```
  
  Then I run the following
  
  ```
  for i in $(cat *.csv); do echo ${i}; /usr/bin/sratoolkit/bin/fastq-dump --gzip ${i} & done
  
  ```
  
  1. Some of my downloads takes a lot of time days. Is there a faster way to get stuff from NCBI?
  2. Are there mirrors of NCBI to accelerate the process?
  3. Why is it so slow?
  
  ---
  
  **EDIT 2024**  
  You might want to look into:
  
  * [`ncbi-acc-download`](https://github.com/kblin/ncbi-acc-download): A CLI tool for efficient downloads from NCBI Entrez.
  * [`NSDPY`](https://nsdpy.readthedocs.io/en/latest/main.html): A Python library for batch downloading with additional features like taxonomic sorting.
  
  **Tags:** phylogenetics
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-04-24. Score: 6
  
  Answers to this question can be found on [biostars](https://www.biostars.org/p/91885/). To give you a short summary:
  
  1. Faster way might be to use the `parallel-fastq-dump`, as suggested in this [answer](https://www.biostars.org/p/91885/#300457). I never tested that tool though, my own experience is that `prefetch` is more stable than `fastq-dump` command.
  2. In that same post it seems that the Japanese mirror seems faster, don't know if that is still valid.
  3. The reason this is never optimized might be that downloading from `SRA` is not a task you'll have to do repeatedly, as described here in this [post](https://www.biostars.org/p/91885/#91895).
  
  > **Comments:**
  > **username_2**: Thanks, comprehensive answer. Is it this site https://www.ddbj.nig.ac.jp/  ?  
  > **username_1**: https://www.ddbj.nig.ac.jp/dra/index-e.html, yes it is the same.  
  
  ---
  
  ### Answer by username_3 on 2019-04-29. Score: 1
  
  You can try `wget` to download SRA files from NCBI server. This seems to be 2x faster in terms of CPU time than `prefetch` and has the advantage of not clogging your `/home/ncbi/public/sra/` directory in case you have space limitation. The tutorial for `wget` downloading method is available at <https://github.com/johanzi/ncbi_tutorial>.
  
  Here my timing of the `prefetch` method:
  
  ```
  time prefetch SRR3300113
  real    3m5.609s
  user    0m6.332s
  sys     0m2.076s
  
  ```
  
  Versus the `wget` method:
  
  ```
  time wget ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR330/SRR3300113/SRR3300113.sra
  real    3m4.696s
  user    0m0.456s
  sys     0m3.332s
  
  ```
  
  That makes a user+sys time of 8.402s for `prefetch` and 3.788s for `wget`. Not so much difference for real time though (about a second).
  
  > **Comments:**
  > **username_4**: Hello username_3, welcome to BIoinformatics.SE. Is there a reason to think that `wget` would be faster? Intuitively I would guess that it's the server that is clogged, then using a different program to query the server does not sound like something that would help. (by the way, my inability to see how it could help does not mean that it does not :-)  
  > **username_3**: Hi username_4, the `wget` process is about 2x faster in terms of CPU time than `prefetch` method based on my test:  
  > **username_4**: `user` + `sys` sort of tells you how long CPUs were active during the `real` time of execution (i.e. for user `real` is what matters, you can check [this](https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1)). So, what I would get out of your nice test is, that the `wget` is a more optimized tool, but both of them are fetching the data from exactly the same place and the limiting factor is the speed of their ftp.  
  > **username_3**: Yes. I would say the main advantage of `wget` is that you can choose where to put your downloaded SRA files while it goes in default into your home directory `/home/user/ncbi/public/sra` with `prefetch`. It can be a problem when you work on a server and you have limited disk usage for your home directory.  
  > **username_5**: You can choose where to put the cache, at least as of a few versions ago, via vdb-config. I agree it's a total pain that it defaults to the home directory instead of the current one, but it's a fixable problem.  
  
  ---
  
  