Answers.Body: <p>I couldn't help but looking a bit deeper into this. There is actually an <a href="https://cran.r-project.org/web/packages/bold/" rel="nofollow noreferrer">R package that uses the BOLD API</a>.</p>
  
  <p>However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a <a href="https://docs.ropensci.org/bold/#large-data" rel="nofollow noreferrer">workaround</a> (downloading sequences separately for each taxonomic Class).</p>
  
  <p>So you should be able to download the marker sequences using the suggested steps:</p>
  
  <pre><code># install packages (if necessary)
  install.packages("bold")
  install.packages("taxize")
  install.packages("seqinr")
  # load packages
  library(tidyverse)
  library(bold)    # API interface to BOLD
  library(taxize)  # for NCBI taxonomy lookup
  library(seqinr)  # for FASTA output
  
  # get class-level taxa within "Animalia" from NCBI taxonomy
  taxa &lt;- downstream("Animalia", db = "ncbi", downto = "class")
  # check if taxa present in BOLD
  checks &lt;- bold_tax_name(taxa<span class="math-container">$Animalia$</span>childtaxa_name)
  taxa_bold &lt;- checks[!is.na(checks<span class="math-container">$taxon),]$</span>taxon
  
  # download sequences from BOLD for each class-level taxon
  sequences &lt;- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %&gt;%
    flatten() %&gt;%
    bind_rows()
  
  # write sequences to file
  write.fasta(
    sequences = as.list(sequences<span class="math-container">$sequence), 
    names = as.list(sequences$</span>id), 
    nbchar = 80, 
    file.out = 'coi5p.fasta')
  </code></pre>
  
  <p>Note that I selected the first tree taxa (<code>taxa_bold[1:3]</code>) for testing, so that it runs quicker. If you want to get all taxa, just remove the <code>[1:3]</code>.</p>
  
  <p>One last importand caveat: In the <a href="https://cran.r-project.org/web/packages/bold/bold.pdf#Rfn.bold.Rul.seq" rel="nofollow noreferrer">help for <code>bold_seq</code></a> it says that</p>
  
  <blockquote>
    <p>Notes from BOLD on the <code>marker</code> param:  "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.</p>
  </blockquote>
  
  <p>So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P <strong>and</strong> ITS markers being present in the database (it affects the e-values, though).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T06:45:06.830Z
Answers.Text: I tried it and when downloading all sequences I receive following warnings: ```Warning messages:
  1: In .f(.x[[i]], ...) :
    the request timed out, see 'If a request times out'
  returning partial output
  2: In .f(.x[[i]], ...) :
    the request timed out, see 'If a request times out'
  returning partial output
  3: In .f(.x[[i]], ...) :
    the request timed out, see 'If a request times out'
  returning partial output```
  
  it downloaded 1012676 sequences, so quiet alot but I don't think that all are downloaded. Again, these warnings seem to be related to some kind of download limitation..
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T07:55:48.150Z
Answers.Text: If there are timeouts one must assume that not all sequences have been downloaded. You could also try going a to a deeper taxonomic rank than class (order?) in case that some classes have so many sequences that it leads to a timeout.
Answers.UserDisplayName: mrhd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T08:37:23.253Z
Answers.Text: I think I will use the ```esearch``` as final tool, as it is more to the point than the BOLD downloads. Thanks for your extensive help regarding the issue, for which I awarded the bounty.
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T09:06:09.073Z
Answers.Text: Thank you @Robvh! If I had to choose for myself I would also go with GenBank as a source.
Answers.UserDisplayName: mrhd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-28T09:01:41.170Z
Answers.Text: @mrhd hey so I took interest in this script. The `bind_rows()` command returns the following error. 
  `Error in bind_rows():
  ! Argument 1 must be a data frame or a named atomic vector.
  Run rlang::last_trace() to see where the error occurred.`
  
  Can you lend a helping hand after all these years?
Answers.UserDisplayName: Nickmofoe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T12:58:06.717Z
Answers.LastActivityDate: 2020-05-04T12:58:06.717Z
Answers.UserDisplayName: mrhd
Answers.Body: <p>A straightforward way of getting the sequences from GenBank is from their website using this URL:
  <a href="https://www.ncbi.nlm.nih.gov/nuccore?term=COI%5BGene%20Name%5D" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]</a></p>
  
  <p>There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".</p>
  
  <p>There are >3 mio hits, so the download could take a while.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T09:41:37.163Z
Answers.Text: Thank for your suggestion, this is certainly a possibility for the Gen Bank data. For now, I see Gen Bank data as a good alternative, but am still looking for a solution to the BOLD database.
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T11:17:06.327Z
Answers.Text: I tried to download the sequences from BOLD with `wget` using the API link you provided above, and got the same stalling behaviour that you described. Maybe they apply some sort of limit per download. I'm not really familliar with BOLD.
Answers.UserDisplayName: mrhd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T11:29:08.403Z
Answers.Text: Thanks for replicating my problem, then I know my computer is not the problem. I will contact them and ask for any limitations. I will post back their reply.
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T06:25:39.377Z
Answers.Text: downloading all COI sequences with the method you provide in this answer also does not work. Around ~150MB download the downloading stops with error "failed-network error". I assume this is also some download limitation as we do not experience any network errors?
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T06:30:42.253Z
Answers.Text: check the update of the original post to see the answer of BOLD regarding the stalling behavior.
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T07:35:31.567Z
Answers.Text: I'm now downloading all COI gen bank sequences with```esearch``` from the ```E-Utilities``` package in the Ubuntu terminal using command: ```esearch -db nucleotide -query "COI[GENE]" | efetch -format fasta > COI_all_genbank.fasta```. This should do the trick.
Answers.UserDisplayName: RobH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T07:56:57.003Z
Answers.Text: That's a quite resourceful idea, @Robvh! Didn't know about that command line tool.
Answers.UserDisplayName: mrhd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T09:39:05.277Z
Answers.LastActivityDate: 2020-05-04T09:39:05.277Z
Answers.UserDisplayName: mrhd
Body: <p>I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: <code>blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt</code>. However this results in errors similar to this post: <a href="https://www.biostars.org/p/359971/" rel="nofollow noreferrer">https://www.biostars.org/p/359971/</a> . </p>
  
  <p>These errors probably appear due to the number of sequences in my file (around 700) and 
  the remote connection is thus interrupted. </p>
  
  <p>I found that a solution would be to use <code>blastn</code> with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.</p>
  
  <p>I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal <code>COI</code> sequences from BOLD.</p>
  
  <p>When entering <code>COI-5P</code> as search term on <a href="http://v4.boldsystems.org/index.php/Public_SearchTerms" rel="nofollow noreferrer">http://v4.boldsystems.org/index.php/Public_SearchTerms</a> I receive error: <code>Your search terms resulted in too many matching terms. Please try again with more specific search criteria.</code>. I could likely download the sequences from all the <code>phyla</code> etc seperately and merge them, but I'd rather just download 1 file.</p>
  
  <p>I also tried to use the API by running: <code>wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P</code>. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.</p>
  
  <p><strong>UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."</strong></p>
  
  <p><strong>Does anyone have a solution to download all COI sequences from BOLD in one file?</strong></p>
  
  <p>I could also download COI sequences from gen bank using the <a href="ftp://ftp.ncbi.nlm.nih.gov/blast/db/" rel="nofollow noreferrer">ftp://ftp.ncbi.nlm.nih.gov/blast/db/</a> URL, but I'm not sure which exact files I need. For <code>16S, 18S,..</code> it is obvious, but not for COI. Any suggestions?</p>
  
  <p>Thanks for the help.</p>
  
Tags: <public-databases><data-download><blastn>
Title: Downloading all COI sequences from BOLD fails
ThreadText: # Downloading all COI sequences from BOLD fails
  
  (Asked by: username_0 on 2020-05-01)
  
  I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: `blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt`. However this results in errors similar to this post: <https://www.biostars.org/p/359971/> .
  
  These errors probably appear due to the number of sequences in my file (around 700) and
  the remote connection is thus interrupted.
  
  I found that a solution would be to use `blastn` with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.
  
  I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal `COI` sequences from BOLD.
  
  When entering `COI-5P` as search term on <http://v4.boldsystems.org/index.php/Public_SearchTerms> I receive error: `Your search terms resulted in too many matching terms. Please try again with more specific search criteria.`. I could likely download the sequences from all the `phyla` etc seperately and merge them, but I'd rather just download 1 file.
  
  I also tried to use the API by running: `wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P`. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.
  
  **UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."**
  
  **Does anyone have a solution to download all COI sequences from BOLD in one file?**
  
  I could also download COI sequences from gen bank using the <ftp://ftp.ncbi.nlm.nih.gov/blast/db/> URL, but I'm not sure which exact files I need. For `16S, 18S,..` it is obvious, but not for COI. Any suggestions?
  
  Thanks for the help.
  
  **Tags:** public-databases, data-download, blastn
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-04. Score: 2
  
  I couldn't help but looking a bit deeper into this. There is actually an [R package that uses the BOLD API](https://cran.r-project.org/web/packages/bold/).
  
  However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a [workaround](https://docs.ropensci.org/bold/#large-data) (downloading sequences separately for each taxonomic Class).
  
  So you should be able to download the marker sequences using the suggested steps:
  
  ```
  # install packages (if necessary)
  install.packages("bold")
  install.packages("taxize")
  install.packages("seqinr")
  # load packages
  library(tidyverse)
  library(bold)    # API interface to BOLD
  library(taxize)  # for NCBI taxonomy lookup
  library(seqinr)  # for FASTA output
  
  # get class-level taxa within "Animalia" from NCBI taxonomy
  taxa <- downstream("Animalia", db = "ncbi", downto = "class")
  # check if taxa present in BOLD
  checks <- bold_tax_name(taxa$Animalia$childtaxa_name)
  taxa_bold <- checks[!is.na(checks$taxon),]$taxon
  
  # download sequences from BOLD for each class-level taxon
  sequences <- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %>%
    flatten() %>%
    bind_rows()
  
  # write sequences to file
  write.fasta(
    sequences = as.list(sequences$sequence), 
    names = as.list(sequences$id), 
    nbchar = 80, 
    file.out = 'coi5p.fasta')
  
  ```
  
  Note that I selected the first tree taxa (`taxa_bold[1:3]`) for testing, so that it runs quicker. If you want to get all taxa, just remove the `[1:3]`.
  
  One last importand caveat: In the [help for `bold_seq`](https://cran.r-project.org/web/packages/bold/bold.pdf#Rfn.bold.Rul.seq) it says that
  
  > Notes from BOLD on the `marker` param: "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.
  
  So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P **and** ITS markers being present in the database (it affects the e-values, though).
  
  > **Comments:**
  > **username_0**: I tried it and when downloading all sequences I receive following warnings: ```Warning messages: 1: In .f(.x[[i]], ...) :   the request timed out, see 'If a request times out' returning partial output 2: In .f(.x[[i]], ...) :   the request timed out, see 'If a request times out' returning partial output 3: In .f(.x[[i]], ...) :   the request timed out, see 'If a request times out' returning partial output```  it downloaded 1012676 sequences, so quiet alot but I don't think that all are downloaded. Again, these warnings seem to be related to some kind of download limitation..  
  > **username_1**: If there are timeouts one must assume that not all sequences have been downloaded. You could also try going a to a deeper taxonomic rank than class (order?) in case that some classes have so many sequences that it leads to a timeout.  
  > **username_0**: I think I will use the ```esearch``` as final tool, as it is more to the point than the BOLD downloads. Thanks for your extensive help regarding the issue, for which I awarded the bounty.  
  > **username_1**: Thank you @Robvh! If I had to choose for myself I would also go with GenBank as a source.  
  > **username_2**: @username_1 hey so I took interest in this script. The `bind_rows()` command returns the following error.  `Error in bind_rows(): ! Argument 1 must be a data frame or a named atomic vector. Run rlang::last_trace() to see where the error occurred.`  Can you lend a helping hand after all these years?  
  
  ---
  
  ### Answer by username_1 on 2020-05-04. Score: 1
  
  A straightforward way of getting the sequences from GenBank is from their website using this URL:
  [https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]](https://www.ncbi.nlm.nih.gov/nuccore?term=COI%5BGene%20Name%5D)
  
  There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".
  
  There are >3 mio hits, so the download could take a while.
  
  > **Comments:**
  > **username_0**: Thank for your suggestion, this is certainly a possibility for the Gen Bank data. For now, I see Gen Bank data as a good alternative, but am still looking for a solution to the BOLD database.  
  > **username_1**: I tried to download the sequences from BOLD with `wget` using the API link you provided above, and got the same stalling behaviour that you described. Maybe they apply some sort of limit per download. I'm not really familliar with BOLD.  
  > **username_0**: Thanks for replicating my problem, then I know my computer is not the problem. I will contact them and ask for any limitations. I will post back their reply.  
  > **username_0**: downloading all COI sequences with the method you provide in this answer also does not work. Around ~150MB download the downloading stops with error "failed-network error". I assume this is also some download limitation as we do not experience any network errors?  
  > **username_0**: check the update of the original post to see the answer of BOLD regarding the stalling behavior.  
  > **username_0**: I'm now downloading all COI gen bank sequences with```esearch``` from the ```E-Utilities``` package in the Ubuntu terminal using command: ```esearch -db nucleotide -query "COI[GENE]" | efetch -format fasta > COI_all_genbank.fasta```. This should do the trick.  
  > **username_1**: That's a quite resourceful idea, @Robvh! Didn't know about that command line tool.  
  
  ---
  
  