Answers.Body: <p>My solution is simply.</p>
  <ol>
  <li>perform a web-based Blastp using the following parameters:</li>
  </ol>
  <ul>
  <li>most dissimilar</li>
  <li>5000 hits (maximum limit)</li>
  <li>Restrict to Archaea (which is what you samples appear to be)*</li>
  </ul>
  <p>If you hit the limit of 5000 results, I would repeat the search for each class or phylum within Archaea. I can't remember if it was re-classified as a kingdom or was just another phylum. Concatenate all hits into a single fasta file.</p>
  <ol start="2">
  <li><p>Reg-ex your extremophiles database (personally I'd use pandas), so that the species name of each bacterium is recovered no other information is present. <em>Or</em> this is an easily accessible value within a hash/database (this is a better solution).</p>
  </li>
  <li><p>Use Biopython to loop through your blast fasta file and filter using <code>re</code> search against the species from your database. If you get a hit save that sequence.</p>
  </li>
  </ol>
  <p>Ideally, the point 2 and 3 would be a pandas merge operation, but looping is easier.</p>
  <p>*, If there are extremophiles which are not Archaea (I don't know much about them), then take each general taxonomic group and repeat the operation for the search criteria &quot;restrict the blastp to ... <em>specific group</em>&quot;.</p>
  <hr />
  <p><strong>Issues</strong> The issues with command line blastp, not via the web, is that access is greatly restricted during US working hours. If you are searching more than 100 hits and carrying out the search during the working day US time - they will not let it through. 100 hits or less is fine however. For &gt;100 hits (e.g. 5000) it's got to be done well outside US working hours/ weekends. Web-based blastp doesn't carry this restriction of access during US working hours. Basically, the API route is deprioritised for the web-based approach. I assume this is to stop bots - albeit of course a bot can automate through NCBI's webpage.</p>
  <p><strong>Note</strong> The original solution by @terdon might easily work as a remote blastp because each query is restricted to &lt;100 - so it will always get through during working hours. What I am saying is if you're using Blast API it needs a carefully defined strategy, else shift to the web for blanket approaches.</p>
  <p>Entrez operations from a commandline do work ok during US working day time - albeit it's got to be coded to deal with &quot;random&quot; server refusals otherwise it will break the code (sporadic queries are refused). This is just my recent experience nothing more.</p>
  <p>Just to be clear - this is potentially a medium sized project. Once the strategy has been decided then specific code that supports this approach can be given in separate questions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T11:35:57.520Z
Answers.Text: I deleted my answer because I couldn't get the entrez query to work well. Too many of the species names were problematic.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T12:21:10.263Z
Answers.Text: @M__ your comment on the Issues is revelation to me as in the past I tried to BLAST  >100 proteins at the same time. And sometime it worked other time it did not. Now it makes perfect sense. I have also submitted a ticket to BLAST team, if they offer another solution I will share.
Answers.UserDisplayName: Supertech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T13:40:25.837Z
Answers.Text: NCBI Entrez not working @terdon ... I've encountered bizarre **non-universal** foreign "link" key issues (I assume they're using SQL). In bacteria this is terrible: there were separate data and link structures present for one protein database and not the other, particularly how they linked to nucleotides ... it's disjointed. As a user I had to write two separate solutions in the code (I wouldn't mind but they never tell you either, got to figure it out). Whats happened is separate teams worked on "biologically the same" problem. The result is a *committee meeting*.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T13:45:07.530Z
Answers.Text: @Supertech ticket to blast support I tried this too. I hope you get better success than I had.  You said "[my] comments ... revelation", thanks, but upvotes and "accepts" are much more welcome than "revelations".
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T11:01:20.963Z
Answers.LastActivityDate: 2023-10-14T13:37:29.497Z
Answers.LastEditDate: 2023-10-14T13:37:29.497Z
Answers.UserDisplayName: M__
Answers.Body: <p>In the interface of <a href="https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastp&amp;PAGE_TYPE=BlastSearch&amp;LINK_LOC=blasthome" rel="nofollow noreferrer">blastp</a>, you can limit the search space to organisms you want to. Yo can do this by introducing name species or Tax IDs, you can add more than one, but I bet you don't wanna introduce the 1,500 one by one.</p>
  <p>The only other way to do this, is by using blastp in its <a href="https://blast.ncbi.nlm.nih.gov/doc/blast-help/downloadblastdata.html" rel="nofollow noreferrer">command-line version</a>. You would need to build a custom database to be used as the searching space, and then run blast on your computer or server using this database. Bulding it is a pain in the ass, as far as I know. You need first to download all the proteins in all organisms in fasta format and then use this compilation as a database (makeblastdb command). I am not aware of a way to download all proteins from various organism given a list of taxonomic IDs, but if any, it has to be NCBI, Uniprot or similar. Then you can use blastp locally together with your custom database.</p>
  <p>But why limiting the search space is such a way? You are biasing your results to what you want to get, instead of finding the stuff that actually is closest to your protein of interest. My suggestion is to use the online version of blastp with one of the various databases available and rather than pre-selecting organisms, you can exclude the ones from which you definitely don't want to get results from, and you can also exclude entire families/genera such as primates (taxid:9443), to name an example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T16:04:29.183Z
Answers.Text: There are many valid reasons why you would want to limit the results. For instance, if you are looking for the protein's homologs in particular taxa, you don't care about what is closest.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T00:11:04.500Z
Answers.Text: If you have noticed, database I want to use contains only (hyper)thermophilic organisms. That is exactly what I want.
Answers.UserDisplayName: Supertech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T00:12:43.433Z
Answers.Text: There should be a way of downloading genomes based on the TaxIDs. I will send a ticket to the NCBI support team. In the past they have been very responsive. Maybe a feature they want to add if is not already possible.
Answers.UserDisplayName: Supertech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T08:48:29.507Z
Answers.Text: @terdon, True, there are such cases or others. I still would suggest to first try to play with the "add/remove organism" boxes in the interface and then parse the blast output in order to search for the organisms of interest.
Answers.UserDisplayName: JRodrigoF
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T15:58:22.000Z
Answers.LastActivityDate: 2023-10-14T08:43:38.267Z
Answers.LastEditDate: 2023-10-14T08:43:38.267Z
Answers.UserDisplayName: JRodrigoF
Body: <p>I have a list of organisms with their full scientific names and Taxonomy IDs, total ~1500. I want to BLAST my protein against this 1500 genome or proteome. How can I do that?</p>
  <p>Here a sample protein that I want to search the following database:
  <a href="http://togodb.org/db/thermobase" rel="nofollow noreferrer">http://togodb.org/db/thermobase</a>
  Please note that database can be downloades as CSV file. I am very comfortable using web tools or programmatic tools such as Perl, Python, bash, or Entrez Programming Utilities.</p>
  <pre><code>&gt;QCC44558.1 chitinase [Halobacterium salinarum] TaxID 641522631
  MPHDRRSYLRTSSAVIASLLAASTPTSAADTPPEWDPDTVYTDGDKATFDGYVWEAKWWTKGDKPGADEW
  GPWTQRRPVDGSPGGPTAAFTVTDTTVEPGTAVTVDASDTTGSVDSYEWAFGDGTTAAGVTASHTYDAAG
  EYTIELTVTTGDGTTDTATQQITATTSPGDDEFKVVGYYPSWKGTDDYDFYPADVPFDQVTDVLYAFLDV
  QPDGTVVLPDSDVDHESLLASFASLKQDRAADTRLKLSIGGWGLSPGFEDAAADQASRERFATTAVDLMR
  TYDFDGIDVDWEHPGPRRGKCECGSAQGPANHVALLETVRDRLNDAEVEDGRTYDLSVANGGSDWNAALI
  NHREVAAVVDDIYMMAYDFTGVWHGTAGLNAPIYGTPPDYPPSGDAQQYTLETTLTIWKEQGYWVDWMEW
  EDHGDPVDDPGTLVLGMPFYGRGCNVENGIWDTFSLSEWQQGDPKYQNDVIPPGTWNDLRGPDDANTGAF
  DYGDLAANYEGADAWTKQRNEQGGVPYLWNDSEGVFISYDDPTSIAAKVELAVAEDLGGVMIWELSQDYD
  GTLLETINQHTP
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-13T15:36:02.017Z
Comments.Text: @M__ Yes you are assessment is correct. I will consider breaking it down. I guess I have to build a local BLAST database first using TaxIDs. I will give try and maybe delete my question.
Comments.UserDisplayName: Supertech
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-13T15:56:29.747Z
Comments.Text: I will propose a different solution over the weekend. It's a lot easier than performing a local blast. My strong advice is to take the easy route. Whilst @terdon and I have done this style of calculation - there's much easier ways. This is because for major repeat projects - this level of coding is worth while - for what appears a one-off blast its not worth it. Also interogating NCBI via its link system is never fun (it's not standardised, i.e. thier databases don't use universal foreign keys). Basically solutions around remote blast only - there's a lot of caveats here and I'll describe them
Comments.UserDisplayName: M__
Tags: <blast><ncbi><blastp><makeblastdb><e-utilities>
Title: How to BLAST a protein against a large set of organisms?
ThreadText: # How to BLAST a protein against a large set of organisms?
  
  (Asked by: username_0 on 2023-10-13)
  
  I have a list of organisms with their full scientific names and Taxonomy IDs, total ~1500. I want to BLAST my protein against this 1500 genome or proteome. How can I do that?
  
  Here a sample protein that I want to search the following database:
  <http://togodb.org/db/thermobase>
  Please note that database can be downloades as CSV file. I am very comfortable using web tools or programmatic tools such as Perl, Python, bash, or Entrez Programming Utilities.
  
  ```
  >QCC44558.1 chitinase [Halobacterium salinarum] TaxID 641522631
  MPHDRRSYLRTSSAVIASLLAASTPTSAADTPPEWDPDTVYTDGDKATFDGYVWEAKWWTKGDKPGADEW
  GPWTQRRPVDGSPGGPTAAFTVTDTTVEPGTAVTVDASDTTGSVDSYEWAFGDGTTAAGVTASHTYDAAG
  EYTIELTVTTGDGTTDTATQQITATTSPGDDEFKVVGYYPSWKGTDDYDFYPADVPFDQVTDVLYAFLDV
  QPDGTVVLPDSDVDHESLLASFASLKQDRAADTRLKLSIGGWGLSPGFEDAAADQASRERFATTAVDLMR
  TYDFDGIDVDWEHPGPRRGKCECGSAQGPANHVALLETVRDRLNDAEVEDGRTYDLSVANGGSDWNAALI
  NHREVAAVVDDIYMMAYDFTGVWHGTAGLNAPIYGTPPDYPPSGDAQQYTLETTLTIWKEQGYWVDWMEW
  EDHGDPVDDPGTLVLGMPFYGRGCNVENGIWDTFSLSEWQQGDPKYQNDVIPPGTWNDLRGPDDANTGAF
  DYGDLAANYEGADAWTKQRNEQGGVPYLWNDSEGVFISYDDPTSIAAKVELAVAEDLGGVMIWELSQDYD
  GTLLETINQHTP
  
  ```
  
  **Tags:** blast, ncbi, blastp, makeblastdb, e-utilities
  
  > **Comments:**
  > **username_0**: @username_1 Yes you are assessment is correct. I will consider breaking it down. I guess I have to build a local BLAST database first using TaxIDs. I will give try and maybe delete my question.  
  > **username_1**: I will propose a different solution over the weekend. It's a lot easier than performing a local blast. My strong advice is to take the easy route. Whilst @username_2 and I have done this style of calculation - there's much easier ways. This is because for major repeat projects - this level of coding is worth while - for what appears a one-off blast its not worth it. Also interogating NCBI via its link system is never fun (it's not standardised, i.e. thier databases don't use universal foreign keys). Basically solutions around remote blast only - there's a lot of caveats here and I'll describe them  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-10-14. Score: 1 | Accepted answer
  
  My solution is simply.
  
  1. perform a web-based Blastp using the following parameters:
  
  * most dissimilar
  * 5000 hits (maximum limit)
  * Restrict to Archaea (which is what you samples appear to be)\*
  
  If you hit the limit of 5000 results, I would repeat the search for each class or phylum within Archaea. I can't remember if it was re-classified as a kingdom or was just another phylum. Concatenate all hits into a single fasta file.
  
  2. Reg-ex your extremophiles database (personally I'd use pandas), so that the species name of each bacterium is recovered no other information is present. *Or* this is an easily accessible value within a hash/database (this is a better solution).
  3. Use Biopython to loop through your blast fasta file and filter using `re` search against the species from your database. If you get a hit save that sequence.
  
  Ideally, the point 2 and 3 would be a pandas merge operation, but looping is easier.
  
  \*, If there are extremophiles which are not Archaea (I don't know much about them), then take each general taxonomic group and repeat the operation for the search criteria "restrict the blastp to ... *specific group*".
  
  ---
  
  **Issues** The issues with command line blastp, not via the web, is that access is greatly restricted during US working hours. If you are searching more than 100 hits and carrying out the search during the working day US time - they will not let it through. 100 hits or less is fine however. For >100 hits (e.g. 5000) it's got to be done well outside US working hours/ weekends. Web-based blastp doesn't carry this restriction of access during US working hours. Basically, the API route is deprioritised for the web-based approach. I assume this is to stop bots - albeit of course a bot can automate through NCBI's webpage.
  
  **Note** The original solution by @username_2 might easily work as a remote blastp because each query is restricted to <100 - so it will always get through during working hours. What I am saying is if you're using Blast API it needs a carefully defined strategy, else shift to the web for blanket approaches.
  
  Entrez operations from a commandline do work ok during US working day time - albeit it's got to be coded to deal with "random" server refusals otherwise it will break the code (sporadic queries are refused). This is just my recent experience nothing more.
  
  Just to be clear - this is potentially a medium sized project. Once the strategy has been decided then specific code that supports this approach can be given in separate questions.
  
  > **Comments:**
  > **username_2**: I deleted my answer because I couldn't get the entrez query to work well. Too many of the species names were problematic.  
  > **username_0**: @username_1 your comment on the Issues is revelation to me as in the past I tried to BLAST  >100 proteins at the same time. And sometime it worked other time it did not. Now it makes perfect sense. I have also submitted a ticket to BLAST team, if they offer another solution I will share.  
  > **username_1**: NCBI Entrez not working @username_2 ... I've encountered bizarre **non-universal** foreign "link" key issues (I assume they're using SQL). In bacteria this is terrible: there were separate data and link structures present for one protein database and not the other, particularly how they linked to nucleotides ... it's disjointed. As a user I had to write two separate solutions in the code (I wouldn't mind but they never tell you either, got to figure it out). Whats happened is separate teams worked on "biologically the same" problem. The result is a *committee meeting*.  
  > **username_1**: @username_0 ticket to blast support I tried this too. I hope you get better success than I had.  You said "[my] comments ... revelation", thanks, but upvotes and "accepts" are much more welcome than "revelations".  
  
  ---
  
  ### Answer by username_3 on 2023-10-13. Score: 1
  
  In the interface of [blastp](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastp&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome), you can limit the search space to organisms you want to. Yo can do this by introducing name species or Tax IDs, you can add more than one, but I bet you don't wanna introduce the 1,500 one by one.
  
  The only other way to do this, is by using blastp in its [command-line version](https://blast.ncbi.nlm.nih.gov/doc/blast-help/downloadblastdata.html). You would need to build a custom database to be used as the searching space, and then run blast on your computer or server using this database. Bulding it is a pain in the ass, as far as I know. You need first to download all the proteins in all organisms in fasta format and then use this compilation as a database (makeblastdb command). I am not aware of a way to download all proteins from various organism given a list of taxonomic IDs, but if any, it has to be NCBI, Uniprot or similar. Then you can use blastp locally together with your custom database.
  
  But why limiting the search space is such a way? You are biasing your results to what you want to get, instead of finding the stuff that actually is closest to your protein of interest. My suggestion is to use the online version of blastp with one of the various databases available and rather than pre-selecting organisms, you can exclude the ones from which you definitely don't want to get results from, and you can also exclude entire families/genera such as primates (taxid:9443), to name an example.
  
  > **Comments:**
  > **username_2**: There are many valid reasons why you would want to limit the results. For instance, if you are looking for the protein's homologs in particular taxa, you don't care about what is closest.  
  > **username_0**: If you have noticed, database I want to use contains only (hyper)thermophilic organisms. That is exactly what I want.  
  > **username_0**: There should be a way of downloading genomes based on the TaxIDs. I will send a ticket to the NCBI support team. In the past they have been very responsive. Maybe a feature they want to add if is not already possible.  
  > **username_3**: @username_2, True, there are such cases or others. I still would suggest to first try to play with the "add/remove organism" boxes in the interface and then parse the blast output in order to search for the organisms of interest.  
  
  ---
  
  