Answers.Body: <p>As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T23:43:22.900Z
Answers.LastActivityDate: 2020-07-11T23:43:22.900Z
Answers.UserDisplayName: An Ignorant Wanderer
Answers.Body: <p>The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.</p>
  <p><strong>In an argument to an EDirect command:</strong></p>
  <pre><code>esearch -db nuccore -query 'some query' -api_key 12345
  </code></pre>
  <p><strong>In a UNIX environment:</strong></p>
  <p>You can either run this just before you are running some edirect commands or add the following line to your <code>.bashrc</code> file.</p>
  <pre><code>export NCBI_API_KEY=12345
  </code></pre>
  <p><strong>In the environment, from a Perl script:</strong></p>
  <pre><code>$ENV{NCBI_API_KEY} = &quot;12345&quot;
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T00:48:42.473Z
Answers.Text: did as you said, yet I still get "too many requests" for quite a few
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T00:55:16.293Z
Answers.Text: Even went as far as setting the sleep time to 10, and the step size for the for loop to 1, and still get too many requests
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T02:13:40.723Z
Answers.Text: I typically don't use Python for this sort of thing. I just run the command in a bash shell. I suspect the reason for the too many requests error is because you are using multithreading. Does your code simultaneously run 10 instances of the `fetch_metadata` function? May be try without the multithreading to see if it works?
Answers.UserDisplayName: vkkodali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T02:19:48.120Z
Answers.Text: yes that worked. Interesting, thanks!
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T00:36:27.757Z
Answers.LastActivityDate: 2020-07-10T00:36:27.757Z
Answers.UserDisplayName: vkkodali
Answers.Body: <p>I had the same problem. I was making a lot of requests and with the thread it blocked very quickly. But I solved the problem with multiprocessing. Pool and <code>imap_unordered</code>. I can even put <code>ThreadPoolExecutor</code> inside the function. I'm making an application in Gradio (hence the <code>gr.Process()</code>).</p>
  <p>Then there are a few subtleties, if you see correctly, there's an infinite loop in process_gene if there's a 429 error or something else. If it's a 429 error it's just that the server doesn't want to because too many requests have been made, if it's another error you should stop and restart the gene or at least analyse the error better. If, for example, it's a 500 error, it's just that the server is down for maintenance, so you might as well try again.</p>
  <p>Then I make packets from my gene list, because when in doubt, I prefer to send in packets of 50 to avoid overloading the various processor cores. And to finish everything off, I use <code>pool.close()</code> and <code>pool.join()</code> to make sure it finishes all the processes and &quot;empties&quot; the cores to send the next group of genes.</p>
  <p>Now, some information about how I tested and what I have concluded. I tried it on a Windows on Windows 10, 8GB RAM, almost 10 year old i5 processor (4 cores) and I tried it on a Mac Mini M2 8GB RAM, it has 8 cores. As you can see in my code (<code>multiprocessing.Pool(processes=multiprocessing.cpu_count())</code>), I used all the cores. With Windows, I manage to extract 90 gene_names per minute (not with the script below, I have another script where each <code>process_gene()</code> makes between 4 and 5 requests in a row + web-scraping, the script below is just an example). With Windows, sometimes I still get error 429 but as there's a loop, I get through it. In theory, the Mac should be 2x faster because it has 8 cores. But as the API limits to 10 requests per second, I'm stuck at 90 gene_names per minute.</p>
  <p>I strongly advise you to use the API key, but you're already doing that.</p>
  <p>I hope this helps :)</p>
  <pre class="lang-py prettyprint-override"><code>import random
  import time
  import requests
  
  def process_gene(gene_id):
      while True:
          url = f&quot;https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&amp;id={gene_id}&amp;retmode=json&amp;rettype=xml&amp;api_key={YOUR_API_KEY}&quot;
          response = requests.get(url)
  
          if response.status_code == 200:
              response_data = response.json()
              try:
                  gene_info = response_data['result'][str(gene_id)]
                  gene_name = gene_info['name']
  
                  print(f&quot;Response 200: Info for {gene_id} {gene_name} retrieved.&quot;)
                  return gene_name, f&quot;Response 200: Info for {gene_id} {gene_name} retrieved.&quot;, gene_id
              
              except Exception as e:
                  print(f&quot;Response 200: Info for {gene_id} not found.&quot;)
                  return &quot;Error 200&quot;, f&quot;Info for {gene_id} not found.&quot;, gene_id
  
          elif response.status_code == 429:
              print(f&quot;Error 429: API rate limit exceeded during get {gene_id} info, try again: {response.text}&quot;)
              time.sleep(random.uniform(0.25, 0.5))
          else:
              print(f&quot;Error {response.status_code}: {response.text}&quot;)
              time.sleep(random.uniform(0.25, 0.5))
      
  
  def your_function(list_gene, progress=gr.Progress()):
      result = []
      result_error = []
      gene_ids = list_gene.strip().split(&quot;\n&quot;)
      total_genes = len(gene_ids)
      progress(0.0, desc=&quot; Extracting... ⚠️ PLEASE WAIT UNTIL COMPLETION&quot;)
  
      gene_groups = [gene_ids[i:i + 50] for i in range(0, len(gene_ids), 50)]
      i = 0
      j = 0
  
      for group in gene_groups:
          j += 1
          with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
              results = pool.imap_unordered(process_gene,[gene_id for gene_id in group])
  
              for result_output, message, gene_id in results:
                  if result_output == &quot;Error 200&quot;:
                      result_error.append(message)
                  else:
                      result.append(result_output)
  
                  i += 1
                  progress(i / total_genes, desc=f&quot; Extracting {gene_id}... {j}/{len(gene_groups)} - {i}/{total_genes}&quot;)
  
              pool.close()
              pool.join()
              
      return result, result_error
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T11:05:36.993Z
Answers.LastActivityDate: 2024-09-09T14:56:13.323Z
Answers.LastEditDate: 2024-09-09T14:56:13.323Z
Answers.UserDisplayName: Julien Minniti
Body: <p>I keep getting a &quot;too many requests error&quot; when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.</p>
  <p>Here is my (Python) code:</p>
  <pre><code>import subprocess
  import concurrent.futures
  import time
  
  bioprojects = [
      &quot;PRJNA644722&quot;,
      &quot;PRJNA644892&quot;
  ]
  
  all_metadata_generators = []
  
  def fetch_metadata(bioproject):
      return subprocess.run(f&quot;esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo&quot;, shell=True, capture_output=True, text=True)
  
  with concurrent.futures.ThreadPoolExecutor() as executor:
      for i in range(0, len(bioprojects), 10):
          all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
          time.sleep(1)
  
  
  for metadata_generator in all_metadata_generators:
      for metadata in metadata_generator:
          print(metadata)
  
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T21:52:07.963Z
Comments.Text: Ahh... this was often rumoured and there whispers on the wind of departmental bans - just rumours. You bang on with the `sleep`, I would just increase it until the signal detection police are happy. If you use a try: exception loop you could even try and attempt to attenuate the `sleep` time to just about the detection limit. Goodluck
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T22:00:46.303Z
Comments.Text: @Michael thing is here `sleep` is technically not even doing anything since I only have 2 bioproject ids. So I can't even do 2 requests in more than one second without problems
Comments.UserDisplayName: An Ignorant Wanderer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T23:02:46.327Z
Comments.Text: where do you have the API key? In your `.bashrc` file? I would run the same command from the command line to check if you still get the 429 error. If you don't then you may have to specify the API key in the python command again.
Comments.UserDisplayName: vkkodali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T23:11:47.843Z
Comments.Text: @vkkodali how do you specify it on the command line? I couldn't find it in the e-utilities documentation
Comments.UserDisplayName: An Ignorant Wanderer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T23:14:42.257Z
Comments.Text: @vkkodali ah I see now that actually the API key was not set. I set it from Biopython (and I'm not using Biopython in my code) so obviously it wasn't going to be used above. Silly me!
Comments.UserDisplayName: An Ignorant Wanderer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-09T23:15:55.587Z
Comments.Text: @vkkodali but how do you pass it to the esearch / efetch commands?
Comments.UserDisplayName: An Ignorant Wanderer
Tags: <python><phylogenetics>
Title: Too many requests error when querying NCBI
ThreadText: # Too many requests error when querying NCBI
  
  (Asked by: username_0 on 2020-07-09)
  
  I keep getting a "too many requests error" when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.
  
  Here is my (Python) code:
  
  ```
  import subprocess
  import concurrent.futures
  import time
  
  bioprojects = [
      "PRJNA644722",
      "PRJNA644892"
  ]
  
  all_metadata_generators = []
  
  def fetch_metadata(bioproject):
      return subprocess.run(f"esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo", shell=True, capture_output=True, text=True)
  
  with concurrent.futures.ThreadPoolExecutor() as executor:
      for i in range(0, len(bioprojects), 10):
          all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
          time.sleep(1)
  
  for metadata_generator in all_metadata_generators:
      for metadata in metadata_generator:
          print(metadata)
  
  ```
  
  **Tags:** python, phylogenetics
  
  > **Comments:**
  > **username_1**: Ahh... this was often rumoured and there whispers on the wind of departmental bans - just rumours. You bang on with the `sleep`, I would just increase it until the signal detection police are happy. If you use a try: exception loop you could even try and attempt to attenuate the `sleep` time to just about the detection limit. Goodluck  
  > **username_0**: @Michael thing is here `sleep` is technically not even doing anything since I only have 2 bioproject ids. So I can't even do 2 requests in more than one second without problems  
  > **username_2**: where do you have the API key? In your `.bashrc` file? I would run the same command from the command line to check if you still get the 429 error. If you don't then you may have to specify the API key in the python command again.  
  > **username_0**: @username_2 how do you specify it on the command line? I couldn't find it in the e-utilities documentation  
  > **username_0**: @username_2 ah I see now that actually the API key was not set. I set it from Biopython (and I'm not using Biopython in my code) so obviously it wasn't going to be used above. Silly me!  
  > **username_0**: @username_2 but how do you pass it to the esearch / efetch commands?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2020-07-11. Score: 2 | Accepted answer
  
  As @username_2 pointed out, the problem turned out to be that you shouldn't use multithreading.
  
  
  ---
  
  ### Answer by username_2 on 2020-07-10. Score: 4
  
  The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.
  
  **In an argument to an EDirect command:**
  
  ```
  esearch -db nuccore -query 'some query' -api_key 12345
  
  ```
  
  **In a UNIX environment:**
  
  You can either run this just before you are running some edirect commands or add the following line to your `.bashrc` file.
  
  ```
  export NCBI_API_KEY=12345
  
  ```
  
  **In the environment, from a Perl script:**
  
  ```
  $ENV{NCBI_API_KEY} = "12345"
  
  ```
  
  > **Comments:**
  > **username_0**: did as you said, yet I still get "too many requests" for quite a few  
  > **username_0**: Even went as far as setting the sleep time to 10, and the step size for the for loop to 1, and still get too many requests  
  > **username_2**: I typically don't use Python for this sort of thing. I just run the command in a bash shell. I suspect the reason for the too many requests error is because you are using multithreading. Does your code simultaneously run 10 instances of the `fetch_metadata` function? May be try without the multithreading to see if it works?  
  > **username_0**: yes that worked. Interesting, thanks!  
  
  ---
  
  ### Answer by username_3 on 2024-09-06. Score: 3
  
  I had the same problem. I was making a lot of requests and with the thread it blocked very quickly. But I solved the problem with multiprocessing. Pool and `imap_unordered`. I can even put `ThreadPoolExecutor` inside the function. I'm making an application in Gradio (hence the `gr.Process()`).
  
  Then there are a few subtleties, if you see correctly, there's an infinite loop in process\_gene if there's a 429 error or something else. If it's a 429 error it's just that the server doesn't want to because too many requests have been made, if it's another error you should stop and restart the gene or at least analyse the error better. If, for example, it's a 500 error, it's just that the server is down for maintenance, so you might as well try again.
  
  Then I make packets from my gene list, because when in doubt, I prefer to send in packets of 50 to avoid overloading the various processor cores. And to finish everything off, I use `pool.close()` and `pool.join()` to make sure it finishes all the processes and "empties" the cores to send the next group of genes.
  
  Now, some information about how I tested and what I have concluded. I tried it on a Windows on Windows 10, 8GB RAM, almost 10 year old i5 processor (4 cores) and I tried it on a Mac Mini M2 8GB RAM, it has 8 cores. As you can see in my code (`multiprocessing.Pool(processes=multiprocessing.cpu_count())`), I used all the cores. With Windows, I manage to extract 90 gene\_names per minute (not with the script below, I have another script where each `process_gene()` makes between 4 and 5 requests in a row + web-scraping, the script below is just an example). With Windows, sometimes I still get error 429 but as there's a loop, I get through it. In theory, the Mac should be 2x faster because it has 8 cores. But as the API limits to 10 requests per second, I'm stuck at 90 gene\_names per minute.
  
  I strongly advise you to use the API key, but you're already doing that.
  
  I hope this helps :)
  
  ```py
  import random
  import time
  import requests
  
  def process_gene(gene_id):
      while True:
          url = f"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&id={gene_id}&retmode=json&rettype=xml&api_key={YOUR_API_KEY}"
          response = requests.get(url)
  
          if response.status_code == 200:
              response_data = response.json()
              try:
                  gene_info = response_data['result'][str(gene_id)]
                  gene_name = gene_info['name']
  
                  print(f"Response 200: Info for {gene_id} {gene_name} retrieved.")
                  return gene_name, f"Response 200: Info for {gene_id} {gene_name} retrieved.", gene_id
              
              except Exception as e:
                  print(f"Response 200: Info for {gene_id} not found.")
                  return "Error 200", f"Info for {gene_id} not found.", gene_id
  
          elif response.status_code == 429:
              print(f"Error 429: API rate limit exceeded during get {gene_id} info, try again: {response.text}")
              time.sleep(random.uniform(0.25, 0.5))
          else:
              print(f"Error {response.status_code}: {response.text}")
              time.sleep(random.uniform(0.25, 0.5))
      
  
  def your_function(list_gene, progress=gr.Progress()):
      result = []
      result_error = []
      gene_ids = list_gene.strip().split("\n")
      total_genes = len(gene_ids)
      progress(0.0, desc=" Extracting... ⚠️ PLEASE WAIT UNTIL COMPLETION")
  
      gene_groups = [gene_ids[i:i + 50] for i in range(0, len(gene_ids), 50)]
      i = 0
      j = 0
  
      for group in gene_groups:
          j += 1
          with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
              results = pool.imap_unordered(process_gene,[gene_id for gene_id in group])
  
              for result_output, message, gene_id in results:
                  if result_output == "Error 200":
                      result_error.append(message)
                  else:
                      result.append(result_output)
  
                  i += 1
                  progress(i / total_genes, desc=f" Extracting {gene_id}... {j}/{len(gene_groups)} - {i}/{total_genes}")
  
              pool.close()
              pool.join()
              
      return result, result_error
  
  ```
  
  
  ---
  
  