Answers.Body: <p>The search term you need is <strong>[gene name]</strong> rather than <em>[title]</em>. For example,</p>
  <pre><code>(Gasterosteus aculeatus[Organism] AND cox1[Gene Name]) OR (Gasterosteus aculeatus[Organism] AND coi[Gene Name]) 
  </code></pre>
  <p>Results in <strong>266</strong> hits which all contain COX1 COI</p>
  <p>Using Daphnia magna ...</p>
  <pre><code>(Daphnia magna[Organism] AND cox1[Gene Name]) OR (Daphnia magna[Organism] AND coi[Gene Name]) 
  </code></pre>
  <p>This results in <strong>775</strong> hits.</p>
  <p><em>Daphina magna</em> comprises <strong>61</strong> mtDNA genomes, you perform the above search and sort by sequence length and set the sequences per page to 200(easy).</p>
  <p>Hope that resolves the issue.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T17:11:36.337Z
Answers.LastActivityDate: 2023-11-11T17:19:14.627Z
Answers.LastEditDate: 2023-11-11T17:19:14.627Z
Answers.UserDisplayName: M__
Answers.Body: <p>There are two main problems I can see in your approach. First, gene names are not standardized across species and the &quot;same&quot; gene (what that means is a whole different discussion) can have different names. For example, human  &quot;Cytochrome c oxidase assembly factor 1&quot; <a href="https://www.genenames.org/data/gene-symbol-report/#!/hgnc_id/HGNC:21868" rel="nofollow noreferrer">has the symbol &quot;COA1&quot;</a>.</p>
  <p>Next, I don't quite understand why you expect the word &quot;complete&quot; to be in the title. That is actually why you don't find the <em>D. magna</em> sequence: it isn't complete, the start codon hasn't been correctly annotated as you can see on the <a href="https://www.ncbi.nlm.nih.gov/gene/24147205/" rel="nofollow noreferrer">relevant NCBI gene page</a>:</p>
  <p><a href="https://i.sstatic.net/7xM2k.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7xM2k.png" alt="Screenshot showing that the start codon is not determined" /></a></p>
  <p>I would instead try something like this:</p>
  <pre><code>((chrM[Chromosome]) OR (chromosome M[Chromosome]) OR (MT[Chromosome]) OR (chromosome MT[Chromosome])) AND ((CO1[Gene Name]) OR (COX1[Gene Name]) OR ((cytochrome[Gene Name]) AND (oxidase[Gene Name]))
  </code></pre>
  <p>or</p>
  <pre><code>((16S[Gene/Protein Name]) and (ribosomal[Gene/Protein Name]))
  </code></pre>
  <p>All that said, trying to rely on gene annotations is rarely the best approach. There just isn't enough standardization. I would instead look at databases that provide lists of homologous genes. For instance, NCBI's <a href="https://www.ncbi.nlm.nih.gov/research/cog/" rel="nofollow noreferrer">Clusters of Orthologous Genes (COGs)</a>, which provides grouped homologous sequences.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T14:21:55.817Z
Answers.Text: Thanks for the tips they are very relevant. But the COG is only a viable option for archea and bacteria so it doesn't suit for me.
Answers.UserDisplayName: Mirko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T14:38:40.373Z
Answers.Text: Ah, crud, sorry @Mirko, I didn't know it was limited to microorganisms. There is also [KOG](https://www.hsls.pitt.edu/obrc/index.php?page=URL1144075392) and [Ortho MCL](https://orthomcl.org/orthomcl/app) or you can use BioMart to extract orthologs etc. My main point here is that trying to use a gene name is unlikely to work well and you should look for dedicated resources instead.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T13:53:43.673Z
Answers.LastActivityDate: 2023-11-13T16:02:07.980Z
Answers.LastEditDate: 2023-11-13T16:02:07.980Z
Answers.UserDisplayName: terdon
Body: <p>I want to retrieve a sequence for many species from the Nucleotide database in NCBI.</p>
  <p>I'm using a command line approach and I have to figure out what is the best query that will return exclusively the region of DNA I am interested in and filter out unwanted noisy sequences.</p>
  <p>I am using a wide range of species (that I have stored as a list of TaxIDs of about 2000 species): it includes small crustaceans, invertebrates, some algae and small vertebrates (reptiles, amphibians and few mammals).</p>
  <p>My final goal is to obtain a phylogenetic tree for all or most of the species.</p>
  <p>I have been suggested to use these genes to create such phylogeny</p>
  <ul>
  <li>CO1 mithocondrial DNA</li>
  <li>16S rRNA</li>
  <li>18S rRNA</li>
  </ul>
  <p>I want to formulate a specific query that will return exclusively those sequences.</p>
  <p>I'm using the GenBank query builder to visually check for the accuracy of my search and when I find a good query I will use it in the API.</p>
  <p>So far I came up with the following queries:</p>
  <ul>
  <li><code>(COX1[Title] OR CO1[Title]) AND complete[Title]</code> --&gt; 63/2000 Species</li>
  <li><code>16S[title] AND complete[title] AND rRNA[title] NOT partial[title]</code> --&gt; 9/2000 species</li>
  <li><code>18S[title] AND complete[title] AND rRNA[title] NOT partial[title]</code> --&gt; 15/2000 species</li>
  </ul>
  <p>As you can see the number of species that I get sequence for is very low compared to the initial 2000 species. I doubt that we have so little available sequences (especially for COX1 that is used for barcoding)</p>
  <p>Can you help me understanding whether my queries are good or not? And if possible suggest a better alternative</p>
  <h3>More Info</h3>
  <p>A subset of 10 of my species of interest is</p>
  <pre><code>Rasbora heteromorpha
  Elasmopus rapax
  Gasterosteus aculeatus
  Palaemonetes pugio
  Catostomus commersoni
  Daphnia magna
  Oryzias latipes
  Xenopus laevis
  Tigriopus japonicus
  Oncorhynchus mykiss
  </code></pre>
  <p>Out of these 10 only 2 have sequences available using the COX1 query</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Name</th>
  <th>SeqID</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Oncorhynchus mykiss</td>
  <td>EU186789.1</td>
  </tr>
  <tr>
  <td>Xenopus laevis</td>
  <td>AB278691.1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>But Daphnia magna is one of the most commonly used organisms in lab and I found <a href="https://www.tandfonline.com/doi/full/10.1080/23802359.2020.1750995" rel="nofollow noreferrer">this paper</a> regarding the complete mitogenome of a specific strain of D. magna thus implying that the complete mitogenome of the species is already known. That means that there must be a way to retrieve all the other species that have full mitogenome but no COX1 gene mentioned in their GenBank title.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T09:52:45.030Z
Comments.Text: What database are you searching? I get 1995 hits for `16S[title] AND complete[title] AND rRNA[title] NOT partial[title]`, across 1430 eukaryotes, 287 bacteria and 178 archaea when using "nucleotide".
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T10:53:21.193Z
Comments.Text: Yes sorry maybe i was not very clear: the numbers on the right are the species that are both in my database and also in the results from Nucleotide. So even if you get 1400 eukariotes only 9 of these are overlapping with the species i work with. But they are mostly lab species and not rare, That is why I find it strange that i don't get many sequences from my search
Comments.UserDisplayName: Mirko
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T13:07:27.367Z
Comments.Text: You seem to be assuming gene names are shared across organisms. It's a reasonable assumption but, sadly, it is absolutely false. Gene names are _not_ shared across species. They can be, sometimes, but not always or even often. For example, there is no `CO1` or `COX1` in human. "Cytochrome c oxidase assembly factor 1" [has the symbol "COA1"](https://www.genenames.org/data/gene-symbol-report/#!/hgnc_id/HGNC:21868).
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-17T14:23:05.810Z
Comments.Text: For anyone having the same type of problem as me (batch COI download) i recommend reading this article as it has proven very useful for me DOI: 10.1111/1755-0998.13756
Comments.UserDisplayName: Mirko
Tags: <phylogenetics><phylogeny><sequence-annotation><genbank><barcode>
Title: What is the best Query to retrieve DNA from NCBI?
ThreadText: # What is the best Query to retrieve DNA from NCBI?
  
  (Asked by: username_0 on 2023-11-08)
  
  I want to retrieve a sequence for many species from the Nucleotide database in NCBI.
  
  I'm using a command line approach and I have to figure out what is the best query that will return exclusively the region of DNA I am interested in and filter out unwanted noisy sequences.
  
  I am using a wide range of species (that I have stored as a list of TaxIDs of about 2000 species): it includes small crustaceans, invertebrates, some algae and small vertebrates (reptiles, amphibians and few mammals).
  
  My final goal is to obtain a phylogenetic tree for all or most of the species.
  
  I have been suggested to use these genes to create such phylogeny
  
  * CO1 mithocondrial DNA
  * 16S rRNA
  * 18S rRNA
  
  I want to formulate a specific query that will return exclusively those sequences.
  
  I'm using the GenBank query builder to visually check for the accuracy of my search and when I find a good query I will use it in the API.
  
  So far I came up with the following queries:
  
  * `(COX1[Title] OR CO1[Title]) AND complete[Title]` --> 63/2000 Species
  * `16S[title] AND complete[title] AND rRNA[title] NOT partial[title]` --> 9/2000 species
  * `18S[title] AND complete[title] AND rRNA[title] NOT partial[title]` --> 15/2000 species
  
  As you can see the number of species that I get sequence for is very low compared to the initial 2000 species. I doubt that we have so little available sequences (especially for COX1 that is used for barcoding)
  
  Can you help me understanding whether my queries are good or not? And if possible suggest a better alternative
  
  ### More Info
  
  A subset of 10 of my species of interest is
  
  ```
  Rasbora heteromorpha
  Elasmopus rapax
  Gasterosteus aculeatus
  Palaemonetes pugio
  Catostomus commersoni
  Daphnia magna
  Oryzias latipes
  Xenopus laevis
  Tigriopus japonicus
  Oncorhynchus mykiss
  
  ```
  
  Out of these 10 only 2 have sequences available using the COX1 query
  
  | Name | SeqID |
  | --- | --- |
  | Oncorhynchus mykiss | EU186789.1 |
  | Xenopus laevis | AB278691.1 |
  
  But Daphnia magna is one of the most commonly used organisms in lab and I found [this paper](https://www.tandfonline.com/doi/full/10.1080/23802359.2020.1750995) regarding the complete mitogenome of a specific strain of D. magna thus implying that the complete mitogenome of the species is already known. That means that there must be a way to retrieve all the other species that have full mitogenome but no COX1 gene mentioned in their GenBank title.
  
  **Tags:** phylogenetics, phylogeny, sequence-annotation, genbank, barcode
  
  > **Comments:**
  > **username_1**: What database are you searching? I get 1995 hits for `16S[title] AND complete[title] AND rRNA[title] NOT partial[title]`, across 1430 eukaryotes, 287 bacteria and 178 archaea when using "nucleotide".  
  > **username_0**: Yes sorry maybe i was not very clear: the numbers on the right are the species that are both in my database and also in the results from Nucleotide. So even if you get 1400 eukariotes only 9 of these are overlapping with the species i work with. But they are mostly lab species and not rare, That is why I find it strange that i don't get many sequences from my search  
  > **username_1**: You seem to be assuming gene names are shared across organisms. It's a reasonable assumption but, sadly, it is absolutely false. Gene names are _not_ shared across species. They can be, sometimes, but not always or even often. For example, there is no `CO1` or `COX1` in human. "Cytochrome c oxidase assembly factor 1" [has the symbol "COA1"](https://www.genenames.org/data/gene-symbol-report/#!/hgnc_id/HGNC:21868).  
  > **username_0**: For anyone having the same type of problem as me (batch COI download) i recommend reading this article as it has proven very useful for me DOI: 10.1111/1755-0998.13756  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-11-11. Score: 1 | Accepted answer
  
  The search term you need is **[gene name]** rather than *[title]*. For example,
  
  ```
  (Gasterosteus aculeatus[Organism] AND cox1[Gene Name]) OR (Gasterosteus aculeatus[Organism] AND coi[Gene Name]) 
  
  ```
  
  Results in **266** hits which all contain COX1 COI
  
  Using Daphnia magna ...
  
  ```
  (Daphnia magna[Organism] AND cox1[Gene Name]) OR (Daphnia magna[Organism] AND coi[Gene Name]) 
  
  ```
  
  This results in **775** hits.
  
  *Daphina magna* comprises **61** mtDNA genomes, you perform the above search and sort by sequence length and set the sequences per page to 200(easy).
  
  Hope that resolves the issue.
  
  
  ---
  
  ### Answer by username_1 on 2023-11-11. Score: 2
  
  There are two main problems I can see in your approach. First, gene names are not standardized across species and the "same" gene (what that means is a whole different discussion) can have different names. For example, human "Cytochrome c oxidase assembly factor 1" [has the symbol "COA1"](https://www.genenames.org/data/gene-symbol-report/#!/hgnc_id/HGNC:21868).
  
  Next, I don't quite understand why you expect the word "complete" to be in the title. That is actually why you don't find the *D. magna* sequence: it isn't complete, the start codon hasn't been correctly annotated as you can see on the [relevant NCBI gene page](https://www.ncbi.nlm.nih.gov/gene/24147205/):
  
  [![Screenshot showing that the start codon is not determined](https://i.sstatic.net/7xM2k.png)](https://i.sstatic.net/7xM2k.png)
  
  I would instead try something like this:
  
  ```
  ((chrM[Chromosome]) OR (chromosome M[Chromosome]) OR (MT[Chromosome]) OR (chromosome MT[Chromosome])) AND ((CO1[Gene Name]) OR (COX1[Gene Name]) OR ((cytochrome[Gene Name]) AND (oxidase[Gene Name]))
  
  ```
  
  or
  
  ```
  ((16S[Gene/Protein Name]) and (ribosomal[Gene/Protein Name]))
  
  ```
  
  All that said, trying to rely on gene annotations is rarely the best approach. There just isn't enough standardization. I would instead look at databases that provide lists of homologous genes. For instance, NCBI's [Clusters of Orthologous Genes (COGs)](https://www.ncbi.nlm.nih.gov/research/cog/), which provides grouped homologous sequences.
  
  > **Comments:**
  > **username_0**: Thanks for the tips they are very relevant. But the COG is only a viable option for archea and bacteria so it doesn't suit for me.  
  > **username_1**: Ah, crud, sorry @username_0, I didn't know it was limited to microorganisms. There is also [KOG](https://www.hsls.pitt.edu/obrc/index.php?page=URL1144075392) and [Ortho MCL](https://orthomcl.org/orthomcl/app) or you can use BioMart to extract orthologs etc. My main point here is that trying to use a gene name is unlikely to work well and you should look for dedicated resources instead.  
  
  ---
  
  