Answers.Body: <p>This is likely a throttling issue - see <a href="https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&amp;PAGE_TYPE=BlastDocs&amp;DOC_TYPE=DeveloperInfo" rel="nofollow noreferrer">here</a> for some details, specifically the one about contacting the NCBI server more than once every 10 seconds.</p>
  <p>You may be able to submit a multiFASTA of all your queries to get around this issue, rather than individual FASTA sequences, or alternatively, add in a <code>sleep 10</code> command between your BLAST commands and at the end of each <code>for</code> loop iteration to throttle the requests.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-07T01:28:40.670Z
Answers.LastActivityDate: 2022-10-07T01:38:51.927Z
Answers.LastEditDate: 2022-10-07T01:38:51.927Z
Answers.UserDisplayName: Scot
Answers.Body: <p>I gave it a whirl using a random bit of SARS-CoV-2</p>
  <pre><code>blastn -query out.fa -db &quot;nt&quot; -remote  -outfmt &quot;6 sacc sseqid sseq evalue bitscore length&quot; &gt;&gt; &quot;results.tsv&quot;
  
  blastn -query out.fa -db &quot;nt&quot; -remote -perc_identity 94 -outfmt &quot;6 sacc sseqid sseq evalue bitscore length&quot; &gt;&gt; &quot;results2.tsv&quot;
  </code></pre>
  <p>Both gave a 500 line output - the limit of the blast specified.</p>
  <p>One possibility is to place a <code>sleep 10</code> line between each <code>blastn</code> command (3 additional lines).</p>
  <p>The rationale is specifications NCBI set:</p>
  <blockquote>
  <p>Do not contact the server more often than <strong>once every 10 seconds</strong>. Do
  not poll for any single RID more often than once a minute. Use the URL
  parameter email and tool, so that the NCBI can contact you if there is
  a problem. Run scripts weekends or between 9 pm and 5 am Eastern time
  on weekdays if more than 50 searches will be submitted.</p>
  </blockquote>
  <hr />
  <p>The suggestions haven't worked, so I'd recommend trying <code>sleep 60</code> between each <code>blastn</code>command on the basis this might be an issue with <code>RID</code>. If that fails the fall back is to run the job overnight US time via <code>cron tab</code>, which I'll post the code if it is required. Thus,</p>
  <blockquote>
  <p>Run scripts weekends or between 9 pm and 5 am Eastern time
  on weekdays if more than 50 searches will be submitted.</p>
  </blockquote>
  <p>The final thing to add is that it is a fairly long job so connectivity might be an issue. If this was the case then it might also be resolved by running the script overnight in the OPs timezone, i.e. via <code>cron</code></p>
  <hr />
  <p><code>sleep 60</code> works but is very slow.</p>
  <ul>
  <li>Agreed one option is to download the database and perform the Blast locally.</li>
  <li>Alternative use <a href="https://github.com/soedinglab/MMseqs2" rel="nofollow noreferrer">mmseqs2</a></li>
  <li>You could perform the Blast on the Cloud, like GCP (much faster).</li>
  <li>Personally I would look at performing over night runs when the network was not in use.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-07T03:04:23.063Z
Answers.Text: FYI @scot posted first ... similar ideas but credit to them.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-07T01:43:10.310Z
Answers.LastActivityDate: 2022-10-08T00:12:46.483Z
Answers.LastEditDate: 2022-10-08T00:12:46.483Z
Answers.UserDisplayName: M__
Body: <p>I am trying to search the NCBI non-redundant database for sequences similar to a few other (~40) sequences that I already have.</p>
  <p>So I've tried running blastn remotely, and looping through multiple queries. I've tried it both as a job (via slurm) and from the command line.</p>
  <p>In addition, for every query, I am trying to run 3 searches with slightly different parameters (in WebBlast, I noticed that I recover more sequences of interest this way).</p>
  <p>I am trying to dump the output data in a single file (stuff like duplicates can be sorted out later).</p>
  <p>This is my code:</p>
  <pre><code>for i in /directory/*.fa;do #this is the directory containing the queries
  j=${i#&quot;/directory/&quot;}
  k=${j%&quot;.fa&quot;} #getting a short version of the name for the log file
  echo &quot;$k&quot; &gt;&gt; &quot;log.log&quot;
  blastn -query &quot;$i&quot; -db &quot;nt&quot; -remote -entrez_query 700:1000[slen] -perc_identity 94 -outfmt &quot;6 sacc sseqid sseq evalue bitscore length&quot; &gt;&gt; &quot;results.tsv&quot;
  blastn -query &quot;$i&quot; -db &quot;nt&quot; -remote -entrez_query 1000:1400[slen] -perc_identity 94 -outfmt &quot;6 sacc sseqid sseq evalue bitscore length&quot; &gt;&gt; &quot;results.tsv&quot;
  blastn -query &quot;$i&quot; -db &quot;nt&quot; -remote -perc_identity 94 -outfmt &quot;6 sacc sseqid sseq evalue bitscore length&quot; &gt;&gt; &quot;results.tsv&quot;
  done
  </code></pre>
  <p>However, it...doesn't seem to work? Either only the first search for the first query is completed, and then the code does nothing else, or nothing happens. (in a rare case, the first two searches worked) There seems to be no clear distinction between 'it works a tiny bit' and 'it doesn't work at all', which leads me to think it might be something outside the code, e.g. connection issues.</p>
  <p>Has anyone faced this before, and would you know how to fix it?</p>
  <p><strong>Update</strong>
  I tried to run a single search as well. As well as adding the waiting period. Right now it doesn't work (same issue, no error message, just sort of appearing to run without results). In the link that Scot provided below, from the NCBI, it was mentioned 'We will move searches of users who submit more than 100 searches in a 24 hour period to a slower queue, or, in extreme cases, will block the requests.' I haven't run &gt;100 searches yesterday (was planning to before I knew this, but that's a different issue).</p>
  <p>I think I'll limit the searches as recommended and try to run it from a different computer later on.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-07T11:59:09.290Z
Comments.Text: Hi @Laura could you have one final try using `sleep 60`? The other idea is to set up a `cron` job and run it over night, I wouldn't try and manually overcome the challenge.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-07T14:28:48.403Z
Comments.Text: @M__ I tried. First search works as normal (finishes in a couple of minutes). The next are...again, very very slow. I left the job running for a bit, and it's taken 2h for now while only on the 4th search.
Comments.UserDisplayName: Laura
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-07T18:54:02.893Z
Comments.Text: Hi @Laura - depending on your network environment (perhaps on a University network), and the details of how NCBI BLAST throttling works, you may be throttled as a group rather than as an individual. Trying this from a different network may help if this is the case.
Comments.UserDisplayName: Scot
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-07T19:02:11.637Z
Comments.Text: And it is beyond the scope of your question, but if you are desiring to run hundreds of BLASTs eventually, I'd look into downloading the BLAST executable and the `nt` database, and running the alignments locally. `nt` is not a small database, so this may be best done using slurm on your cluster, but if you can limit the alignments to a taxonomic segment of `nt`, you may be able to do this locally using just that taxonomic slice of `nt`.
Comments.UserDisplayName: Scot
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-08T00:03:51.777Z
Comments.Text: Answered in the post
Comments.UserDisplayName: M__
Tags: <blast><bash><loop>
Title: Remote blastn - what is breaking my (bash) loop?
ThreadText: # Remote blastn - what is breaking my (bash) loop?
  
  (Asked by: username_0 on 2022-10-06)
  
  I am trying to search the NCBI non-redundant database for sequences similar to a few other (~40) sequences that I already have.
  
  So I've tried running blastn remotely, and looping through multiple queries. I've tried it both as a job (via slurm) and from the command line.
  
  In addition, for every query, I am trying to run 3 searches with slightly different parameters (in WebBlast, I noticed that I recover more sequences of interest this way).
  
  I am trying to dump the output data in a single file (stuff like duplicates can be sorted out later).
  
  This is my code:
  
  ```
  for i in /directory/*.fa;do #this is the directory containing the queries
  j=${i#"/directory/"}
  k=${j%".fa"} #getting a short version of the name for the log file
  echo "$k" >> "log.log"
  blastn -query "$i" -db "nt" -remote -entrez_query 700:1000[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
  blastn -query "$i" -db "nt" -remote -entrez_query 1000:1400[slen] -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
  blastn -query "$i" -db "nt" -remote -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
  done
  
  ```
  
  However, it...doesn't seem to work? Either only the first search for the first query is completed, and then the code does nothing else, or nothing happens. (in a rare case, the first two searches worked) There seems to be no clear distinction between 'it works a tiny bit' and 'it doesn't work at all', which leads me to think it might be something outside the code, e.g. connection issues.
  
  Has anyone faced this before, and would you know how to fix it?
  
  **Update**
  I tried to run a single search as well. As well as adding the waiting period. Right now it doesn't work (same issue, no error message, just sort of appearing to run without results). In the link that username_2 provided below, from the NCBI, it was mentioned 'We will move searches of users who submit more than 100 searches in a 24 hour period to a slower queue, or, in extreme cases, will block the requests.' I haven't run >100 searches yesterday (was planning to before I knew this, but that's a different issue).
  
  I think I'll limit the searches as recommended and try to run it from a different computer later on.
  
  **Tags:** blast, bash, loop
  
  > **Comments:**
  > **username_1**: Hi @username_0 could you have one final try using `sleep 60`? The other idea is to set up a `cron` job and run it over night, I wouldn't try and manually overcome the challenge.  
  > **username_0**: @username_1 I tried. First search works as normal (finishes in a couple of minutes). The next are...again, very very slow. I left the job running for a bit, and it's taken 2h for now while only on the 4th search.  
  > **username_2**: Hi @username_0 - depending on your network environment (perhaps on a University network), and the details of how NCBI BLAST throttling works, you may be throttled as a group rather than as an individual. Trying this from a different network may help if this is the case.  
  > **username_2**: And it is beyond the scope of your question, but if you are desiring to run hundreds of BLASTs eventually, I'd look into downloading the BLAST executable and the `nt` database, and running the alignments locally. `nt` is not a small database, so this may be best done using slurm on your cluster, but if you can limit the alignments to a taxonomic segment of `nt`, you may be able to do this locally using just that taxonomic slice of `nt`.  
  > **username_1**: Answered in the post  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-10-07. Score: 2 | Accepted answer
  
  I gave it a whirl using a random bit of SARS-CoV-2
  
  ```
  blastn -query out.fa -db "nt" -remote  -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results.tsv"
  
  blastn -query out.fa -db "nt" -remote -perc_identity 94 -outfmt "6 sacc sseqid sseq evalue bitscore length" >> "results2.tsv"
  
  ```
  
  Both gave a 500 line output - the limit of the blast specified.
  
  One possibility is to place a `sleep 10` line between each `blastn` command (3 additional lines).
  
  The rationale is specifications NCBI set:
  
  > Do not contact the server more often than **once every 10 seconds**. Do
  > not poll for any single RID more often than once a minute. Use the URL
  > parameter email and tool, so that the NCBI can contact you if there is
  > a problem. Run scripts weekends or between 9 pm and 5 am Eastern time
  > on weekdays if more than 50 searches will be submitted.
  
  ---
  
  The suggestions haven't worked, so I'd recommend trying `sleep 60` between each `blastn`command on the basis this might be an issue with `RID`. If that fails the fall back is to run the job overnight US time via `cron tab`, which I'll post the code if it is required. Thus,
  
  > Run scripts weekends or between 9 pm and 5 am Eastern time
  > on weekdays if more than 50 searches will be submitted.
  
  The final thing to add is that it is a fairly long job so connectivity might be an issue. If this was the case then it might also be resolved by running the script overnight in the OPs timezone, i.e. via `cron`
  
  ---
  
  `sleep 60` works but is very slow.
  
  * Agreed one option is to download the database and perform the Blast locally.
  * Alternative use [mmseqs2](https://github.com/soedinglab/MMseqs2)
  * You could perform the Blast on the Cloud, like GCP (much faster).
  * Personally I would look at performing over night runs when the network was not in use.
  
  > **Comments:**
  > **username_1**: FYI @username_2 posted first ... similar ideas but credit to them.  
  
  ---
  
  ### Answer by username_2 on 2022-10-07. Score: 2
  
  This is likely a throttling issue - see [here](https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=DeveloperInfo) for some details, specifically the one about contacting the NCBI server more than once every 10 seconds.
  
  You may be able to submit a multiFASTA of all your queries to get around this issue, rather than individual FASTA sequences, or alternatively, add in a `sleep 10` command between your BLAST commands and at the end of each `for` loop iteration to throttle the requests.
  
  
  ---
  
  