Answers.Body: <p>While I haven't used your exact method above, using the BLAST command line tools return the desired number of target sequences</p>
  <pre><code>blastp  -query query.fa \
          -task blastp-short \
          -db nr \
          -max_target_seqs 5000 \
          -outfmt 6 \
          -out query.out \
          -remote
  </code></pre>
  <p>query.fa</p>
  <pre><code>&gt;mytest
  MSILANKDTR
  </code></pre>
  <p>The resulting <code>query.out</code> file does contain 5000 unique hits.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T15:34:31.957Z
Answers.Text: I'll mark this as "accepted" because this is a good solution given the original question I asked (I tried this approach, but didn't state it in the question). The full explanation is given below and the above answer should be read within the context of my explanation for future reference. **Note** unless the other Mods over-rule my "accept" please keep this *comment* here to provide future readers with the full solution and thanks to @scot
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T06:07:52.520Z
Answers.LastActivityDate: 2023-09-22T06:07:52.520Z
Answers.UserDisplayName: Scot
Answers.Body: <p>Right here's the response from NCBI,</p>
  <blockquote>
  <p>Please send your future blast related questions to blast-help@ncbi.nlm.nih.gov directly. The gb-admin alias is for submitters with questions related to their sequence deposition/submission.</p>
  </blockquote>
  <blockquote>
  <p>For your issue, you should look at the following two parameters:
  DESCRIPTIONS  Number of descriptions to print (applies to HTML and Text)  Integer Put,Get Integer greater than zero.
  ALIGNMENTS    Number of alignments to print (applies to HTML and Text)    Integer Put,Get Integer greater than zero.</p>
  </blockquote>
  <blockquote>
  <p>Do please note that asking for this level of hits saved/returned could be problematic for the search. They could make the search exceeds the cpu time allocation and lead to error.</p>
  </blockquote>
  <p>I tried this ... it was the same as &quot;HITLIST_SIZE&quot;.</p>
  <p>I figured the answer ...</p>
  <p><strong>Why its happening?</strong> The API uses a default parameter for the *<em>maximum number of hits</em>. It's imposing a limit on the number of hits returned, but this wasn't stated in the documentation of the parameters.</p>
  <p><strong>Whats the limit?</strong> It's &gt;70 but &lt;= 100. I suspect the limit is 100 hits. Thus anything above this its set at 100.</p>
  <p><strong>Summary</strong> The reason why the API is working and other approaches simply time out is NCBI is imposing an upper limit on the number of hits for the API and the Blast servers are not permitting requests above this threshold at peak time. Thus because other approaches are not imposing the restrictions on the NCBI server, the API gets through, whilst the other methods (blastp directly and Biopython) are timing out.</p>
  <p><strong>NCBI are recommending the API is because it is automating a 100 hit cut-off and the servers are deprioritising limits beyond this</strong></p>
  <p>I have filed a note of advice to NCBI because it should be clear in the documentation there's an upper limit on the HITLIST_SIZE parameter. It took me at least a full half day to figure this out (which isn't cool for limited documentation).</p>
  <hr />
  <p><strong>Proof of concept (untested)</strong> Thus, using @Scott's solution</p>
  <pre><code>blastp  -query query.fa \
          -task blastp-short \
          -db nr \
          -max_target_seqs 100 \
          -outfmt 6 \
          -out query.out \
          -remote
  </code></pre>
  <p>This will work with the <code>-max_target_seqs 100</code> and thats all the API is doing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T15:33:21.397Z
Answers.LastActivityDate: 2023-09-22T18:14:45.070Z
Answers.LastEditDate: 2023-09-22T18:14:45.070Z
Answers.UserDisplayName: M__
Body: <p><strong>Question and Background</strong> I'm using NCBI's Blast Perl API to interrogate Genbank for example <code>blastp</code>. I don't get the throughput using their point and click web stuff.</p>
  <p>I tried Biopython's <code>Bio.Blast import NCBIWWW</code> but its under access restrictions from the host, notably attempting to use it during the day (works sporadically). The API however gets through and runs as fast as a web search ... that's good.</p>
  <p>Here's an example of the command I'm running:</p>
  <pre><code>CMD=Put&amp;PROGRAM=blastp&amp;DATABASE=nr&amp;HITLIST_SIZE=2500&amp;ALIGNMENTS=2500&amp;QUERY=%3Emytest%0AMSILANKDTR%0A
  </code></pre>
  <p>The above query gives default Blast output. What I'm trying to do is increase the number of hit recovered to the maximum (5000). The options for this are <code>HITLIST_SIZE</code>, <code>ALIGNMENTS</code>, <code>DESCRIPTIONS</code>. In the above example I'm attempting a <strong>hit_size</strong> of 2500 ... but the same output is returned as for default. More generally  I can't get any of the &quot;number of hits/descriptions/alignments returned&quot; options to work, the default is returned each time.</p>
  <p>Has anybody solved this?</p>
  <p>Typical options for the API are given below</p>
  <p><strong>Typical options</strong></p>
  <pre><code>Parameter   Definition  Type    CMD Allowed values
  QUERY       Search query    String  Put *   Accession, GI, or FASTA.
  DATABASE    BLAST database  String  Put *   Database from appendix 2 or one uploaded to blastdb_custom (see appendix 4)
  PROGRAM     BLAST program   String  Put *   One of blastn, blastp, blastx, tblastn, tblastx. To enable megablast, use PROGRAM=blastn&amp;MEGABLAST=on.
  FILTER      Low complexity filtering    String  Put F to disable. T or L to enable. Prepend “m” for mask at lookup (e.g., mL)
  FORMAT_TYPE Report type String  Put, Get    HTML, Text, XML, XML2, JSON2, or Tabular. HTML is the default.
  EXPECT      Expect value    Double  Put Number greater than zero.
  HITLIST_SIZE    Number of databases sequences to keep   Integer Put,Get Integer greater than zero.
  DESCRIPTIONS    Number of descriptions to print (applies to HTML and Text)  Integer Put,Get Integer greater than zero.
  ALIGNMENTS      Number of alignments to print (applies to HTML and Text)    Integer Put,Get Integer greater than zero.
  </code></pre>
  <hr />
  <p>Just to feedback on the answer and comment</p>
  <ul>
  <li><p>Using <code>-remote</code> by locally installing blast as recommended by @Scott. This is a valid answer so definitely upvoted and recommended its further upvoted. I had tried this but not mentioned it in the question (really sorry), but it times out due to traffic issues. The API overcomes traffic issues. Again, @Scott's is a valid solution (considering I'd not explained I'd tried it).</p>
  </li>
  <li><p>More details ...</p>
  </li>
  </ul>
  <p>The Perl API is here <a href="https://blast.ncbi.nlm.nih.gov/docs/web_blast.pl" rel="nofollow noreferrer">https://blast.ncbi.nlm.nih.gov/docs/web_blast.pl</a></p>
  <p>Line 95 below ...</p>
  <pre><code><span class="math-container">$args = "CMD=Put&amp;PROGRAM=$</span>program&amp;DATABASE=<span class="math-container">$database&amp;QUERY=" . 
  $</span>encoded_query;
  </code></pre>
  <p>This is changed to,</p>
  <pre><code>$args = &quot;CMD=Put&amp;PROGRAM=$program&amp;DATABASE=$database&amp;HITLIST_SIZE=$maxhits&amp;QUERY=&quot; . $encoded_query;
  </code></pre>
  <p>Also need to add an additional line at line 73</p>
  <pre><code>$maxhits = shift;
  </code></pre>
  <p>From the command line,</p>
  <pre><code>perl ./web-blast.pl blastp nr 2500 ./example.fa
  </code></pre>
  <p>(Note, The variable order is switched in the program).</p>
  <p>example.fa - e.g.</p>
  <pre><code>&gt;rleguminosarum
  WGFDGSSTQQAEGRSSDCVLKPVAIYPDPARTNGALVMCEVMMPDGVTPHASNARATILDDEDAWFGFEQ
  EYFFYQNGRPLGFPEQGYPAPQGPYYTGVGYSNVGDVAREIVEEHLDLCLAAGINHEGINAEVAKGQWEF
  QIFGKGSKKAADQIWMARYLLQRLTEKYGIDI
  </code></pre>
  <p>I'm proficient in Perl5, i.e. there's no error in changes to the code in the API, albeit my expertise is object oriented and high performance Python.</p>
  <hr />
  <p><strong>Limit on the request</strong> ... I'm testing that now.</p>
  <p>First results <em>eeek</em> @terdon might be right ... :-(</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-22T08:11:43.373Z
Comments.Text: Can you give us a minimal example of the actual code that we can test on our local machines, please? I wonder if it could be a limit in individual post requests and if you might need to set the downloading of the results in some kind of loop to get N at a time.
Comments.UserDisplayName: terdon
Tags: <blast><perl><api><parameters>
Title: Remote NCBI's Blast Perl API: maximising hits returned via command line
ThreadText: # Remote NCBI's Blast Perl API: maximising hits returned via command line
  
  (Asked by: username_0 on 2023-09-20)
  
  **Question and Background** I'm using NCBI's Blast Perl API to interrogate Genbank for example `blastp`. I don't get the throughput using their point and click web stuff.
  
  I tried Biopython's `Bio.Blast import NCBIWWW` but its under access restrictions from the host, notably attempting to use it during the day (works sporadically). The API however gets through and runs as fast as a web search ... that's good.
  
  Here's an example of the command I'm running:
  
  ```
  CMD=Put&PROGRAM=blastp&DATABASE=nr&HITLIST_SIZE=2500&ALIGNMENTS=2500&QUERY=%3Emytest%0AMSILANKDTR%0A
  
  ```
  
  The above query gives default Blast output. What I'm trying to do is increase the number of hit recovered to the maximum (5000). The options for this are `HITLIST_SIZE`, `ALIGNMENTS`, `DESCRIPTIONS`. In the above example I'm attempting a **hit\_size** of 2500 ... but the same output is returned as for default. More generally I can't get any of the "number of hits/descriptions/alignments returned" options to work, the default is returned each time.
  
  Has anybody solved this?
  
  Typical options for the API are given below
  
  **Typical options**
  
  ```
  Parameter   Definition  Type    CMD Allowed values
  QUERY       Search query    String  Put *   Accession, GI, or FASTA.
  DATABASE    BLAST database  String  Put *   Database from appendix 2 or one uploaded to blastdb_custom (see appendix 4)
  PROGRAM     BLAST program   String  Put *   One of blastn, blastp, blastx, tblastn, tblastx. To enable megablast, use PROGRAM=blastn&MEGABLAST=on.
  FILTER      Low complexity filtering    String  Put F to disable. T or L to enable. Prepend “m” for mask at lookup (e.g., mL)
  FORMAT_TYPE Report type String  Put, Get    HTML, Text, XML, XML2, JSON2, or Tabular. HTML is the default.
  EXPECT      Expect value    Double  Put Number greater than zero.
  HITLIST_SIZE    Number of databases sequences to keep   Integer Put,Get Integer greater than zero.
  DESCRIPTIONS    Number of descriptions to print (applies to HTML and Text)  Integer Put,Get Integer greater than zero.
  ALIGNMENTS      Number of alignments to print (applies to HTML and Text)    Integer Put,Get Integer greater than zero.
  
  ```
  
  ---
  
  Just to feedback on the answer and comment
  
  * Using `-remote` by locally installing blast as recommended by @Scott. This is a valid answer so definitely upvoted and recommended its further upvoted. I had tried this but not mentioned it in the question (really sorry), but it times out due to traffic issues. The API overcomes traffic issues. Again, @Scott's is a valid solution (considering I'd not explained I'd tried it).
  * More details ...
  
  The Perl API is here <https://blast.ncbi.nlm.nih.gov/docs/web_blast.pl>
  
  Line 95 below ...
  
  ```
  $args = "CMD=Put&PROGRAM=$program&DATABASE=$database&QUERY=" . 
  $encoded_query;
  
  ```
  
  This is changed to,
  
  ```
  $args = "CMD=Put&PROGRAM=$program&DATABASE=$database&HITLIST_SIZE=$maxhits&QUERY=" . $encoded_query;
  
  ```
  
  Also need to add an additional line at line 73
  
  ```
  $maxhits = shift;
  
  ```
  
  From the command line,
  
  ```
  perl ./web-blast.pl blastp nr 2500 ./example.fa
  
  ```
  
  (Note, The variable order is switched in the program).
  
  example.fa - e.g.
  
  ```
  >rleguminosarum
  WGFDGSSTQQAEGRSSDCVLKPVAIYPDPARTNGALVMCEVMMPDGVTPHASNARATILDDEDAWFGFEQ
  EYFFYQNGRPLGFPEQGYPAPQGPYYTGVGYSNVGDVAREIVEEHLDLCLAAGINHEGINAEVAKGQWEF
  QIFGKGSKKAADQIWMARYLLQRLTEKYGIDI
  
  ```
  
  I'm proficient in Perl5, i.e. there's no error in changes to the code in the API, albeit my expertise is object oriented and high performance Python.
  
  ---
  
  **Limit on the request** ... I'm testing that now.
  
  First results *eeek* @username_1 might be right ... :-(
  
  **Tags:** blast, perl, api, parameters
  
  > **Comments:**
  > **username_1**: Can you give us a minimal example of the actual code that we can test on our local machines, please? I wonder if it could be a limit in individual post requests and if you might need to set the downloading of the results in some kind of loop to get N at a time.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-09-22. Score: 2 | Accepted answer
  
  While I haven't used your exact method above, using the BLAST command line tools return the desired number of target sequences
  
  ```
  blastp  -query query.fa \
          -task blastp-short \
          -db nr \
          -max_target_seqs 5000 \
          -outfmt 6 \
          -out query.out \
          -remote
  
  ```
  
  query.fa
  
  ```
  >mytest
  MSILANKDTR
  
  ```
  
  The resulting `query.out` file does contain 5000 unique hits.
  
  > **Comments:**
  > **username_0**: I'll mark this as "accepted" because this is a good solution given the original question I asked (I tried this approach, but didn't state it in the question). The full explanation is given below and the above answer should be read within the context of my explanation for future reference. **Note** unless the other Mods over-rule my "accept" please keep this *comment* here to provide future readers with the full solution and thanks to @username_2  
  
  ---
  
  ### Answer by username_0 on 2023-09-22. Score: 0
  
  Right here's the response from NCBI,
  
  > Please send your future blast related questions to blast-help@ncbi.nlm.nih.gov directly. The gb-admin alias is for submitters with questions related to their sequence deposition/submission.
  
  > For your issue, you should look at the following two parameters:
  > DESCRIPTIONS Number of descriptions to print (applies to HTML and Text) Integer Put,Get Integer greater than zero.
  > ALIGNMENTS Number of alignments to print (applies to HTML and Text) Integer Put,Get Integer greater than zero.
  
  > Do please note that asking for this level of hits saved/returned could be problematic for the search. They could make the search exceeds the cpu time allocation and lead to error.
  
  I tried this ... it was the same as "HITLIST\_SIZE".
  
  I figured the answer ...
  
  **Why its happening?** The API uses a default parameter for the \**maximum number of hits*. It's imposing a limit on the number of hits returned, but this wasn't stated in the documentation of the parameters.
  
  **Whats the limit?** It's >70 but <= 100. I suspect the limit is 100 hits. Thus anything above this its set at 100.
  
  **Summary** The reason why the API is working and other approaches simply time out is NCBI is imposing an upper limit on the number of hits for the API and the Blast servers are not permitting requests above this threshold at peak time. Thus because other approaches are not imposing the restrictions on the NCBI server, the API gets through, whilst the other methods (blastp directly and Biopython) are timing out.
  
  **NCBI are recommending the API is because it is automating a 100 hit cut-off and the servers are deprioritising limits beyond this**
  
  I have filed a note of advice to NCBI because it should be clear in the documentation there's an upper limit on the HITLIST\_SIZE parameter. It took me at least a full half day to figure this out (which isn't cool for limited documentation).
  
  ---
  
  **Proof of concept (untested)** Thus, using @Scott's solution
  
  ```
  blastp  -query query.fa \
          -task blastp-short \
          -db nr \
          -max_target_seqs 100 \
          -outfmt 6 \
          -out query.out \
          -remote
  
  ```
  
  This will work with the `-max_target_seqs 100` and thats all the API is doing.
  
  
  ---
  
  