Answers.Body: <p>Your script works as expected on the example you gave. Running your script results in a <code>protein_interactionswithseqs.csv</code> with lines like:</p>
  <pre><code>$ head -n2 protein_interactionswithseqs.csv 
  Query_ENSP,Query_Name,Partner_Name,Combined_Score,Query_Sequence,Partner_Sequence
  9606.ENSP00000000233,ARF5,RAC2,0.4,MGLTVSALFSRIFGKKQMRILMVGLDAAGKTTILYKLKLGEIVTTIPTIGFNVETVEYKNICFTVWDVGGQDKIRPLWRHYFQNTQGLIFVVDSNDRERVQESADELQKMLQEDELRDAVLLVFANKQDMPNAMPVSELTDKLGLQHLRSRTWYVQATCATQGTGLYDGLDWLSHELSKR,MQAIKCVVVGDGAVGKTCLLISYTTNAFPGEYIPTVPGRPCALLQDAEAGLSSDVEGACWCQTQEGRMTGEAPAPPWPALRGPHHS
  </code></pre>
  <p>However, this is horribly, horribly inefficient! You are fetching the same sequence multiple times. With your example data, the protein ARF5 is actually downloaded 19 times, meaning this will take 18 times more time than needed. This is very likely the cause for the timeout error you describe. Now, it might be that your sequences are too many to be stored in RAM (but I would still try it), so one way around this is to download the sequences first, and then write the mappings. Something like this:</p>
  <pre><code>import pandas as pd
  import requests
  import sys
  requests.packages.urllib3.disable_warnings()
  
  
  def write_protein_seq_file(protein_name, protein_seq):
      file_name = &quot;seqs/&quot; + protein_name
      seq_file = open(file_name, &quot;w&quot;)
      data = f&quot;{protein_seq}\n&quot;
      seq_file.write(data)
      seq_file.close
  
  def read_protein_seq_file(protein_name):
      seq_file = open(&quot;seqs/&quot; + protein_name, &quot;r&quot;)
      data = seq_file.read()
      seq_file.close
      return data.rstrip()
  
  # Function to fetch protein sequence from UniProt
  def fetch_protein_sequence_from_uniprot(protein_name):
      #Declaring UniProt API
      uniport_api_url = f&quot;https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&amp;format=fasta&amp;size=1&quot;
      response = requests.get(uniport_api_url,verify=False)
  
      # Parse the response to extract sequence
      sequence = &quot;&quot;
      if response.ok:
          lines = response.text.split(&quot;\n&quot;)
          for line in lines:
              if not line.startswith(&quot;&gt;&quot;):  # Exclude header lines
                  sequence += line
      return sequence
  
  # Reading the input CSV file
  input_csv = &quot;all_protein_interactions.csv&quot;
  chunksize = 1000  # Batch size
  
  # Create an iterator to read the CSV in chunks
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  # Initialize an empty list to store modified chunks
  modified_chunks = []
  
  # Get all sequences
  for chunk in csv_reader:
      for protein in set(list(chunk.iloc[:, 1]) + list(chunk.iloc[:, 2])):
          protein_seq = fetch_protein_sequence_from_uniprot(protein) 
          write_protein_seq_file(protein,protein_seq)
  
  print(&quot;Finished fetching sequences&quot;, file=sys.stderr)     
  ## Now, process the file again and write the new file
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  for chunk in csv_reader:
  
      # Fetch protein sequences for query and partner names
      query_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 1]]
      partner_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 2]]
  
      # Add fetched sequences as new columns in the chunk
      chunk['Query_Sequence'] = query_sequences
      chunk['Partner_Sequence'] = partner_sequences
  
      # Append the modified chunk to the list
      modified_chunks.append(chunk)
  
  # Concatenate all modified chunks into a single DataFrame
  result_df = pd.concat(modified_chunks, ignore_index=True)
  
  # Save the updated DataFrame back to CSV
  result_df.to_csv(&quot;protein_interactionswithseqs.csv&quot;, index=False) 
  </code></pre>
  <p>First, create a directory named <code>seqs</code>, then run the script above. It will process the csv file twice. Once to get all sequences mentioned in the file and save each one in a file called <code>seqs/$sequence_name</code> (where <code>$sequence_name</code> is replaced by the actual name of each protein. Then, it goes through the file a second time, but instead of the very time consuming download, it now fetches the sequence from the file stored on your computer. Once you are satisfied this works, you can delete the <code>seqs</code> directory.</p>
  <p>Note that I don't have access to a Windows machine, so this was tested on Linux. Also, I am not a python coder, so I am sure this ins't very good python and there will be improvements to be made, but even on the toy example you gave, this speeds things up significantly:</p>
  <pre><code>$ time your_script.py 
  
  real    0m10.186s
  user    0m0.998s
  sys     0m0.076s
  
  $ time my_script.py
  Finished fetching sequences
  
  real    0m3.180s
  user    0m0.456s
  sys     0m0.066s
  </code></pre>
  <p>As you can see above, even on the tiny example with only 19 data lines, your script took 10 seconds while mine only took 3. So try this approach (or, better, take my example and make it more pythonic, add some error checking, some try/except blocks, make sure your data are correct) and you might find the timeout error is no longer an issue. You should certainly find that the whole thing takes far less than three <em>hours</em>(!) to run.</p>
  <p>First of all, that is a really, really inefficient way of doing it. Using your example data, you are getting the same protein, ARF5, 19 times. This makes your program 18 times slower than it needs to be. You should modify the script to download each protein <em>once</em> and then write them all out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T14:23:37.917Z
Answers.Text: Your example script is not taking any file and where to download the all sequences
Answers.UserDisplayName: Anas Jamshed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T16:04:02.893Z
Answers.Text: @AnasJamshed what do you mean? It is taking the exact same file as your version and in the exact same way. The sequences will be downloaded into the `seqs` directory which, as I explain in my answer, you need to create.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T14:04:46.817Z
Answers.LastActivityDate: 2024-01-07T14:04:46.817Z
Answers.UserDisplayName: terdon
Body: <p>I have 193000 protein interactions in CSV named <code>all_proteininteractions.csv</code> which have query protein name in the 2nd column and partner protein in the 3rd like this:</p>
  <pre><code>Query_ENSP,Query_Name,Partner_Name,Combined_Score
  9606.ENSP00000000233,ARF5,RAC2,0.4
  9606.ENSP00000000233,ARF5,RAC2,0.4
  9606.ENSP00000000233,ARF5,PPP2CB,0.408
  9606.ENSP00000000233,ARF5,PPP2CB,0.408
  9606.ENSP00000000233,ARF5,R3HDML,0.413
  9606.ENSP00000000233,ARF5,R3HDML,0.413
  9606.ENSP00000000233,ARF5,PLEK,0.422
  9606.ENSP00000000233,ARF5,PLEK,0.422
  9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
  9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
  9606.ENSP00000000233,ARF5,AP3B1,0.553
  9606.ENSP00000000233,ARF5,AP3B1,0.553
  9606.ENSP00000000233,ARF5,CYTH4,0.579
  9606.ENSP00000000233,ARF5,CYTH4,0.579
  9606.ENSP00000000233,ARF5,RABAC1,0.658
  9606.ENSP00000000233,ARF5,RABAC1,0.658
  9606.ENSP00000000233,ARF5,YKT6,0.672
  9606.ENSP00000000233,ARF5,YKT6,0.672
  9606.ENSP00000000233,ARF5,BET1,0.679
  </code></pre>
  <p>I am using this Python code to fetch sequences of each protein and store them in a CSV file:</p>
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  import requests
  import math
  
  requests.packages.urllib3.disable_warnings()
  
  # Function to fetch protein sequence from UniProt
  def fetch_protein_sequence_from_uniprot(protein_name):
      #Declaring UniProt API
      uniport_api_url = f&quot;https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&amp;format=fasta&amp;size=1&quot;
      response = requests.get(uniport_api_url,verify=False)
  
      # Parse the response to extract sequence
      sequence = &quot;&quot;
      if response.ok:
          lines = response.text.split(&quot;\n&quot;)
          for line in lines:
              if not line.startswith(&quot;&gt;&quot;):  # Exclude header lines
                  sequence += line
      return sequence
  
  # Reading the input CSV file
  input_csv = &quot;all_protein_interactions.csv&quot;
  chunksize = 1000  # Batch size
  
  # Create an iterator to read the CSV in chunks
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  # Initialize an empty list to store modified chunks
  modified_chunks = []
  
  # Process each chunk
  for chunk in csv_reader:
      # Fetch protein sequences for query and partner names
      query_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 1]]
      partner_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 2]]
  
      # Add fetched sequences as new columns in the chunk
      chunk['Query_Sequence'] = query_sequences
      chunk['Partner_Sequence'] = partner_sequences
  
      # Append the modified chunk to the list
      modified_chunks.append(chunk)
  
  # Concatenate all modified chunks into a single DataFrame
  result_df = pd.concat(modified_chunks, ignore_index=True)
  
  # Save the updated DataFrame back to CSV
  result_df.to_csv(&quot;protein_interactionswithseqs.csv&quot;, index=False)
  </code></pre>
  <p>First, it was throwing SSLError, so I made <code>verify=False</code>. But now, after running the script for 3 hours it's showing following error:</p>
  <pre><code>ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  </code></pre>
  <p>How can I remove this connection aborted error to fetch 386000 sequences?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-07T12:45:09.243Z
Comments.Text: You also should add `print` statements in various places, to see where the problem is. Has the script managed to finish the `for chunk` loop? Where exactly is it failing? And isn't your loop trying to get the same protein multiple times? You seem to have the same name in the 2nd column for several lines, so why get that same protein multiple times?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-07T12:46:00.903Z
Comments.Text: Thanks. I am using  Windows 10 with Intel(R) Core(TM) i7-3770 CPU and 12 GB ram.
Comments.UserDisplayName: Anas Jamshed
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-07T12:53:07.677Z
Comments.Text: I modified the text as well. I have similar proteins but I want to fetch them all and store in 5th and 6th column
Comments.UserDisplayName: Anas Jamshed
Tags: <python><uniprot>
Title: Fetching Protein Sequences through UniPROT gives ConnectionError
ThreadText: # Fetching Protein Sequences through UniPROT gives ConnectionError
  
  (Asked by: username_0 on 2024-01-07)
  
  I have 193000 protein interactions in CSV named `all_proteininteractions.csv` which have query protein name in the 2nd column and partner protein in the 3rd like this:
  
  ```
  Query_ENSP,Query_Name,Partner_Name,Combined_Score
  9606.ENSP00000000233,ARF5,RAC2,0.4
  9606.ENSP00000000233,ARF5,RAC2,0.4
  9606.ENSP00000000233,ARF5,PPP2CB,0.408
  9606.ENSP00000000233,ARF5,PPP2CB,0.408
  9606.ENSP00000000233,ARF5,R3HDML,0.413
  9606.ENSP00000000233,ARF5,R3HDML,0.413
  9606.ENSP00000000233,ARF5,PLEK,0.422
  9606.ENSP00000000233,ARF5,PLEK,0.422
  9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
  9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
  9606.ENSP00000000233,ARF5,AP3B1,0.553
  9606.ENSP00000000233,ARF5,AP3B1,0.553
  9606.ENSP00000000233,ARF5,CYTH4,0.579
  9606.ENSP00000000233,ARF5,CYTH4,0.579
  9606.ENSP00000000233,ARF5,RABAC1,0.658
  9606.ENSP00000000233,ARF5,RABAC1,0.658
  9606.ENSP00000000233,ARF5,YKT6,0.672
  9606.ENSP00000000233,ARF5,YKT6,0.672
  9606.ENSP00000000233,ARF5,BET1,0.679
  
  ```
  
  I am using this Python code to fetch sequences of each protein and store them in a CSV file:
  
  ```py
  import pandas as pd
  import requests
  import math
  
  requests.packages.urllib3.disable_warnings()
  
  # Function to fetch protein sequence from UniProt
  def fetch_protein_sequence_from_uniprot(protein_name):
      #Declaring UniProt API
      uniport_api_url = f"https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&format=fasta&size=1"
      response = requests.get(uniport_api_url,verify=False)
  
      # Parse the response to extract sequence
      sequence = ""
      if response.ok:
          lines = response.text.split("\n")
          for line in lines:
              if not line.startswith(">"):  # Exclude header lines
                  sequence += line
      return sequence
  
  # Reading the input CSV file
  input_csv = "all_protein_interactions.csv"
  chunksize = 1000  # Batch size
  
  # Create an iterator to read the CSV in chunks
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  # Initialize an empty list to store modified chunks
  modified_chunks = []
  
  # Process each chunk
  for chunk in csv_reader:
      # Fetch protein sequences for query and partner names
      query_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 1]]
      partner_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 2]]
  
      # Add fetched sequences as new columns in the chunk
      chunk['Query_Sequence'] = query_sequences
      chunk['Partner_Sequence'] = partner_sequences
  
      # Append the modified chunk to the list
      modified_chunks.append(chunk)
  
  # Concatenate all modified chunks into a single DataFrame
  result_df = pd.concat(modified_chunks, ignore_index=True)
  
  # Save the updated DataFrame back to CSV
  result_df.to_csv("protein_interactionswithseqs.csv", index=False)
  
  ```
  
  First, it was throwing SSLError, so I made `verify=False`. But now, after running the script for 3 hours it's showing following error:
  
  ```
  ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  
  ```
  
  How can I remove this connection aborted error to fetch 386000 sequences?
  
  **Tags:** python, uniprot
  
  > **Comments:**
  > **username_1**: You also should add `print` statements in various places, to see where the problem is. Has the script managed to finish the `for chunk` loop? Where exactly is it failing? And isn't your loop trying to get the same protein multiple times? You seem to have the same name in the 2nd column for several lines, so why get that same protein multiple times?  
  > **username_0**: Thanks. I am using  Windows 10 with Intel(R) Core(TM) i7-3770 CPU and 12 GB ram.  
  > **username_0**: I modified the text as well. I have similar proteins but I want to fetch them all and store in 5th and 6th column  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-07. Score: 1
  
  Your script works as expected on the example you gave. Running your script results in a `protein_interactionswithseqs.csv` with lines like:
  
  ```
  $ head -n2 protein_interactionswithseqs.csv 
  Query_ENSP,Query_Name,Partner_Name,Combined_Score,Query_Sequence,Partner_Sequence
  9606.ENSP00000000233,ARF5,RAC2,0.4,MGLTVSALFSRIFGKKQMRILMVGLDAAGKTTILYKLKLGEIVTTIPTIGFNVETVEYKNICFTVWDVGGQDKIRPLWRHYFQNTQGLIFVVDSNDRERVQESADELQKMLQEDELRDAVLLVFANKQDMPNAMPVSELTDKLGLQHLRSRTWYVQATCATQGTGLYDGLDWLSHELSKR,MQAIKCVVVGDGAVGKTCLLISYTTNAFPGEYIPTVPGRPCALLQDAEAGLSSDVEGACWCQTQEGRMTGEAPAPPWPALRGPHHS
  
  ```
  
  However, this is horribly, horribly inefficient! You are fetching the same sequence multiple times. With your example data, the protein ARF5 is actually downloaded 19 times, meaning this will take 18 times more time than needed. This is very likely the cause for the timeout error you describe. Now, it might be that your sequences are too many to be stored in RAM (but I would still try it), so one way around this is to download the sequences first, and then write the mappings. Something like this:
  
  ```
  import pandas as pd
  import requests
  import sys
  requests.packages.urllib3.disable_warnings()
  
  def write_protein_seq_file(protein_name, protein_seq):
      file_name = "seqs/" + protein_name
      seq_file = open(file_name, "w")
      data = f"{protein_seq}\n"
      seq_file.write(data)
      seq_file.close
  
  def read_protein_seq_file(protein_name):
      seq_file = open("seqs/" + protein_name, "r")
      data = seq_file.read()
      seq_file.close
      return data.rstrip()
  
  # Function to fetch protein sequence from UniProt
  def fetch_protein_sequence_from_uniprot(protein_name):
      #Declaring UniProt API
      uniport_api_url = f"https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&format=fasta&size=1"
      response = requests.get(uniport_api_url,verify=False)
  
      # Parse the response to extract sequence
      sequence = ""
      if response.ok:
          lines = response.text.split("\n")
          for line in lines:
              if not line.startswith(">"):  # Exclude header lines
                  sequence += line
      return sequence
  
  # Reading the input CSV file
  input_csv = "all_protein_interactions.csv"
  chunksize = 1000  # Batch size
  
  # Create an iterator to read the CSV in chunks
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  # Initialize an empty list to store modified chunks
  modified_chunks = []
  
  # Get all sequences
  for chunk in csv_reader:
      for protein in set(list(chunk.iloc[:, 1]) + list(chunk.iloc[:, 2])):
          protein_seq = fetch_protein_sequence_from_uniprot(protein) 
          write_protein_seq_file(protein,protein_seq)
  
  print("Finished fetching sequences", file=sys.stderr)     
  ## Now, process the file again and write the new file
  csv_reader = pd.read_csv(input_csv, chunksize=chunksize)
  
  for chunk in csv_reader:
  
      # Fetch protein sequences for query and partner names
      query_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 1]]
      partner_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 2]]
  
      # Add fetched sequences as new columns in the chunk
      chunk['Query_Sequence'] = query_sequences
      chunk['Partner_Sequence'] = partner_sequences
  
      # Append the modified chunk to the list
      modified_chunks.append(chunk)
  
  # Concatenate all modified chunks into a single DataFrame
  result_df = pd.concat(modified_chunks, ignore_index=True)
  
  # Save the updated DataFrame back to CSV
  result_df.to_csv("protein_interactionswithseqs.csv", index=False) 
  
  ```
  
  First, create a directory named `seqs`, then run the script above. It will process the csv file twice. Once to get all sequences mentioned in the file and save each one in a file called `seqs/$sequence_name` (where `$sequence_name` is replaced by the actual name of each protein. Then, it goes through the file a second time, but instead of the very time consuming download, it now fetches the sequence from the file stored on your computer. Once you are satisfied this works, you can delete the `seqs` directory.
  
  Note that I don't have access to a Windows machine, so this was tested on Linux. Also, I am not a python coder, so I am sure this ins't very good python and there will be improvements to be made, but even on the toy example you gave, this speeds things up significantly:
  
  ```
  $ time your_script.py 
  
  real    0m10.186s
  user    0m0.998s
  sys     0m0.076s
  
  $ time my_script.py
  Finished fetching sequences
  
  real    0m3.180s
  user    0m0.456s
  sys     0m0.066s
  
  ```
  
  As you can see above, even on the tiny example with only 19 data lines, your script took 10 seconds while mine only took 3. So try this approach (or, better, take my example and make it more pythonic, add some error checking, some try/except blocks, make sure your data are correct) and you might find the timeout error is no longer an issue. You should certainly find that the whole thing takes far less than three *hours*(!) to run.
  
  First of all, that is a really, really inefficient way of doing it. Using your example data, you are getting the same protein, ARF5, 19 times. This makes your program 18 times slower than it needs to be. You should modify the script to download each protein *once* and then write them all out.
  
  > **Comments:**
  > **username_0**: Your example script is not taking any file and where to download the all sequences  
  > **username_1**: @username_0 what do you mean? It is taking the exact same file as your version and in the exact same way. The sequences will be downloaded into the `seqs` directory which, as I explain in my answer, you need to create.  
  
  ---
  
  