Answers.Body: <p>I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a>, and pipe to standard *nix utilities:</p>
  <pre><code>FastaToTbl file.fa | sort -u | TblToFasta &gt; file.uniq.fa
  </code></pre>
  <p>Or, if you don't have GNU <code>sort</code>:</p>
  <pre><code>FastaToTbl file.fa | sort | uniq | TblToFasta &gt; file.uniq.fa
  </code></pre>
  <p>Alternatively, you can also do something like this:</p>
  <pre><code>awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
  </code></pre>
  <p>For example, given this input file:</p>
  <pre><code>$ cat file.fa
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  </code></pre>
  <p>You can do:</p>
  <pre><code>$ awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  </code></pre>
  <p>Both of the above approaches have the advantage of not needing to store the entire file in memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-24T21:19:46.530Z
Answers.Text: There is also an application at UCSC utilities that converts Fasta to Tab separated file. It's called `faToTab` and can be downloaded from http://hgdownload.soe.ucsc.edu/admin/exe/
Answers.UserDisplayName: Supertech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-24T11:23:35.720Z
Answers.LastActivityDate: 2022-02-24T12:09:38.957Z
Answers.LastEditDate: 2022-02-24T12:09:38.957Z
Answers.UserDisplayName: terdon
Answers.Body: <p>I would use a <code>dict</code> so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N<sup>2</sup>). When you create an entry in the <code>dict</code>, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:</p>
  <pre><code>from Bio import SeqIO
      
  genes = {}
  fasta_file = sys.argv[1]
  for entry in SeqIO.parse( fasta_file, &quot;fasta&quot; ):
      genes[ entry.id ] = entry.seq
      
  </code></pre>
  <p>Then you can write a new, unique fasta file:</p>
  <pre><code>with open('unique_genes.fasta', 'w') as out:
      for key,value in genes.items():
          out.write( '&gt;%s\n%s\n'%(key,value) )
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T19:29:37.060Z
Answers.LastActivityDate: 2021-01-28T19:29:37.060Z
Answers.UserDisplayName: Sevy
Answers.Body: <pre><code>from Bio import SeqIO
  import sys
  import re 
  
  unique = []
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
      if len(myfile) &gt; 250:
          gene_id = myfile.id
          if gene.id not in unique:
              unique.append(gene.id)
              mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
              print (&quot;&gt;&quot;+mylist.group(1))
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-27T15:55:26.173Z
Answers.Text: Yeah, I would do something along these lines too. Though, I would use a set or dict insead of list if the fasta file is big...
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-28T11:46:17.773Z
Answers.LastActivityDate: 2022-02-23T21:36:32.557Z
Answers.LastEditDate: 2022-02-23T21:36:32.557Z
Answers.UserDisplayName: pippo1980
Answers.Body: <p>Perhaps itertools might be an option? From the itertools page / moreitertools project
  <a href="https://docs.python.org/3/library/itertools.html#itertools-recipes" rel="nofollow noreferrer">https://docs.python.org/3/library/itertools.html#itertools-recipes</a></p>
  <pre class="lang-py prettyprint-override"><code>def unique_everseen(iterable, key=None):
      &quot;List unique elements, preserving order. Remember all elements ever seen.&quot;
      # unique_everseen('AAAABBBCCDAABBB') --&gt; A B C D
      # unique_everseen('ABBCcAD', str.lower) --&gt; A B C D
      seen = set()
      seen_add = seen.add
      if key is None:
          for element in filterfalse(seen.__contains__, iterable):
              seen_add(element)
              yield element
      else:
          for element in iterable:
              k = key(element)
              if k not in seen:
                  seen_add(k)
                  yield element
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T08:57:04.337Z
Answers.LastActivityDate: 2021-01-28T08:57:04.337Z
Answers.UserDisplayName: Mladen Rasic
Answers.Body: <p>I use the following code. It doesn't only remove repeated sequences but similar sequences (with a quite simplistic approach).
  I use it just sporadically and I guess it can be largely depurated. Remove the debug prints if these annoy you.</p>
  <pre><code>import sys
  import requests 
  import shlex, subprocess
  import random
  from Bio import SeqIO
  import editdistance
  
  if len(sys.argv)&lt;3:
      print(&quot;Uso: &quot;+sys.argv[0]+&quot; input_fasta output_fasta&quot;)
      sys.exit(0)
  
  savefile = sys.argv[2]
  content_file = sys.argv[1]
  
  class Sequence:
      id=&quot;&quot;
      sequence=&quot;&quot;
  
  def are_similar(seq1, seq2):
      l1 = len(seq1.sequence)
      l2 = len(seq2.sequence)
      if abs(l1-l2) &lt; l1/40:
          d = editdistance.eval(seq1.sequence[0:250], seq2.sequence[0:250])
          #d = editdistance.eval(seq1.sequence, seq2.sequence)
          print(seq1.id+&quot; &quot;+seq2.id+&quot; -&gt; &quot;+str(d))
          #if d &lt; l1/20:
          if d &lt; 50:
              return True
      return False
  
  def find_repeats(sequences, seq, repeated):
      b=False
      for seq2 in sequences:
          if(b and not seq2.id in repeated):
              if are_similar(seq, seq2):
                  repeated.add(seq2.id)        
          if seq2.id==seq.id:
              b=True
  
  sequences=[]
  
  res=&quot;&quot;
  
  for record in SeqIO.parse(content_file, &quot;fasta&quot;):
      seq=Sequence()
      seq.id=record.id
      seq.sequence=record.seq
      sequences.append(seq)
  
  repeated_sequences=set()
  for seq in sequences:
      find_repeats(sequences, seq, repeated_sequences)
  
  print(repeated_sequences)
  
  def mySortFunc(e):
      return len(e.sequence)
  
  sequences.sort(key=mySortFunc)
  
  for seq in sequences:
      b= seq.id in repeated_sequences
      if not b:
          res+=&quot;&gt;&quot;+seq.id+&quot;\n&quot;
          res+=str(seq.sequence)+&quot;\n&quot;
  
  output_file = open(savefile, &quot;w&quot;)
  
  output_file.write(res)
  
  output_file.close()
  </code></pre>
  <p>EDIT: I see that's not exactly what OP is asking ...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T11:45:42.850Z
Answers.LastActivityDate: 2021-01-29T18:24:15.830Z
Answers.LastEditDate: 2021-01-29T18:24:15.830Z
Answers.UserDisplayName: juanjo75es
Answers.Body: <p>Your question isn't well formatted, for example list is used as a variable name where it should be reserved as a keyword...</p>
  <pre><code>print (&quot;&gt;&quot;+list.group(1))
  </code></pre>
  <p>However, the answer you're after is going to be something like:</p>
  <pre><code>print (&quot;&gt;&quot;+set(name_of_list.group(1)))
  </code></pre>
  <p>Anytime you want to get the unique elements of a list, just wrap it in set()</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T03:33:54.757Z
Answers.LastActivityDate: 2021-01-28T03:33:54.757Z
Answers.UserDisplayName: Liam McIntyre
Body: <p>I am using the following command:</p>
  <pre><code>from Bio import SeqIO
  import sys
  import re 
  
  
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
  if len(myfile.seq) &gt; 250:
      gene_id = myfile.id
      mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
      print (&quot;&gt;&quot;+mylist.group(1)) 
  </code></pre>
  <p>And this is providing me with duplicates of the same gene:</p>
  <pre><code>&gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_bTr423_r3_h2
  &gt;H149xcV_kN893_r3_h2
  &gt;H149xcV_DNp021_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_SRt424234_r3_h2
  &gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_Fge342_r3_h2
  </code></pre>
  <p>How can I reformat my command so that I only receive unique gene id's:</p>
  <pre><code>&gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_bTr423_r3_h2
  &gt;H149xcV_kN893_r3_h2
  &gt;H149xcV_DNp021_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_SRt424234_r3_h2
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T00:21:36.387Z
Comments.Text: Why do you need to use python for this?
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T00:23:16.810Z
Comments.Text: I am using `nano` within commandline to write my python command. @RamRS
Comments.UserDisplayName: andnowmywatchbegins
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T02:28:16.413Z
Comments.Text: I'm asking **why**, not **how**. There are command line tools that de-dup fasta, so why reinvent the wheel?
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T02:46:29.597Z
Comments.Text: @RamRS The only command-line tools that I am aware of are `sort`|`uniq` but I am not sure as to how I would use it? I must apologize, I am kind of new to bioinformatics, so I am still getting the hang of it. I know I can output my python into another file, but is there a way to obtain the unique id's afterwards? Thanks!
Comments.UserDisplayName: andnowmywatchbegins
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T09:47:51.473Z
Comments.Text: You can use  `seqkit rmdup` instead reinvent the wheel. https://bioinf.shenwei.me/seqkit/usage/#rmdup
Comments.UserDisplayName: zorbax
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-27T15:47:01.317Z
Comments.Text: @zorbax I think that would be a good answer...
Comments.UserDisplayName: Kamil S Jaron
Tags: <fasta><python>
Title: How to remove duplicates from a fasta file using python
ThreadText: # How to remove duplicates from a fasta file using python
  
  (Asked by: username_0 on 2020-11-23)
  
  I am using the following command:
  
  ```
  from Bio import SeqIO
  import sys
  import re 
  
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, "fasta"):
  if len(myfile.seq) > 250:
      gene_id = myfile.id
      mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
      print (">"+mylist.group(1)) 
  
  ```
  
  And this is providing me with duplicates of the same gene:
  
  ```
  >H149xcV_Fge342_r3_h2
  >H149xcV_bTr423_r3_h2
  >H149xcV_kN893_r3_h2
  >H149xcV_DNp021_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_SRt424234_r3_h2
  >H149xcV_Fge342_r3_h2
  >H149xcV_Fge342_r3_h2
  
  ```
  
  How can I reformat my command so that I only receive unique gene id's:
  
  ```
  >H149xcV_Fge342_r3_h2
  >H149xcV_bTr423_r3_h2
  >H149xcV_kN893_r3_h2
  >H149xcV_DNp021_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_SRt424234_r3_h2
  
  ```
  
  **Tags:** fasta, python
  
  > **Comments:**
  > **username_1**: Why do you need to use python for this?  
  > **username_0**: I am using `nano` within commandline to write my python command. @username_1  
  > **username_1**: I'm asking **why**, not **how**. There are command line tools that de-dup fasta, so why reinvent the wheel?  
  > **username_0**: @username_1 The only command-line tools that I am aware of are `sort`|`uniq` but I am not sure as to how I would use it? I must apologize, I am kind of new to bioinformatics, so I am still getting the hang of it. I know I can output my python into another file, but is there a way to obtain the unique id's afterwards? Thanks!  
  > **username_2**: You can use  `seqkit rmdup` instead reinvent the wheel. https://bioinf.shenwei.me/seqkit/usage/#rmdup  
  > **username_3**: @username_2 I think that would be a good answer...  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_4 on 2022-02-24. Score: 2
  
  I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts [I have posted before](https://bioinformatics.stackexchange.com/a/2651/298), and pipe to standard \*nix utilities:
  
  ```
  FastaToTbl file.fa | sort -u | TblToFasta > file.uniq.fa
  
  ```
  
  Or, if you don't have GNU `sort`:
  
  ```
  FastaToTbl file.fa | sort | uniq | TblToFasta > file.uniq.fa
  
  ```
  
  Alternatively, you can also do something like this:
  
  ```
  awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'
  
  ```
  
  For example, given this input file:
  
  ```
  $ cat file.fa
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  
  ```
  
  You can do:
  
  ```
  $ awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  
  ```
  
  Both of the above approaches have the advantage of not needing to store the entire file in memory.
  
  > **Comments:**
  > **username_5**: There is also an application at UCSC utilities that converts Fasta to Tab separated file. It's called `faToTab` and can be downloaded from http://hgdownload.soe.ucsc.edu/admin/exe/  
  
  ---
  
  ### Answer by username_6 on 2021-01-28. Score: 1
  
  I would use a `dict` so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N2). When you create an entry in the `dict`, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:
  
  ```
  from Bio import SeqIO
      
  genes = {}
  fasta_file = sys.argv[1]
  for entry in SeqIO.parse( fasta_file, "fasta" ):
      genes[ entry.id ] = entry.seq
      
  
  ```
  
  Then you can write a new, unique fasta file:
  
  ```
  with open('unique_genes.fasta', 'w') as out:
      for key,value in genes.items():
          out.write( '>%s\n%s\n'%(key,value) )
  
  ```
  
  
  ---
  
  ### Answer by username_7 on 2020-12-28. Score: 1
  
  ```
  from Bio import SeqIO
  import sys
  import re 
  
  unique = []
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, "fasta"):
      if len(myfile) > 250:
          gene_id = myfile.id
          if gene.id not in unique:
              unique.append(gene.id)
              mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
              print (">"+mylist.group(1))
  
  ```
  
  > **Comments:**
  > **username_3**: Yeah, I would do something along these lines too. Though, I would use a set or dict insead of list if the fasta file is big...  
  
  ---
  
  ### Answer by username_8 on 2021-01-28. Score: 1
  
  Perhaps itertools might be an option? From the itertools page / moreitertools project
  <https://docs.python.org/3/library/itertools.html#itertools-recipes>
  
  ```py
  def unique_everseen(iterable, key=None):
      "List unique elements, preserving order. Remember all elements ever seen."
      # unique_everseen('AAAABBBCCDAABBB') --> A B C D
      # unique_everseen('ABBCcAD', str.lower) --> A B C D
      seen = set()
      seen_add = seen.add
      if key is None:
          for element in filterfalse(seen.__contains__, iterable):
              seen_add(element)
              yield element
      else:
          for element in iterable:
              k = key(element)
              if k not in seen:
                  seen_add(k)
                  yield element
  
  ```
  
  
  ---
  
  ### Answer by username_9 on 2021-01-28. Score: 0
  
  I use the following code. It doesn't only remove repeated sequences but similar sequences (with a quite simplistic approach).
  I use it just sporadically and I guess it can be largely depurated. Remove the debug prints if these annoy you.
  
  ```
  import sys
  import requests 
  import shlex, subprocess
  import random
  from Bio import SeqIO
  import editdistance
  
  if len(sys.argv)<3:
      print("Uso: "+sys.argv[0]+" input_fasta output_fasta")
      sys.exit(0)
  
  savefile = sys.argv[2]
  content_file = sys.argv[1]
  
  class Sequence:
      id=""
      sequence=""
  
  def are_similar(seq1, seq2):
      l1 = len(seq1.sequence)
      l2 = len(seq2.sequence)
      if abs(l1-l2) < l1/40:
          d = editdistance.eval(seq1.sequence[0:250], seq2.sequence[0:250])
          #d = editdistance.eval(seq1.sequence, seq2.sequence)
          print(seq1.id+" "+seq2.id+" -> "+str(d))
          #if d < l1/20:
          if d < 50:
              return True
      return False
  
  def find_repeats(sequences, seq, repeated):
      b=False
      for seq2 in sequences:
          if(b and not seq2.id in repeated):
              if are_similar(seq, seq2):
                  repeated.add(seq2.id)        
          if seq2.id==seq.id:
              b=True
  
  sequences=[]
  
  res=""
  
  for record in SeqIO.parse(content_file, "fasta"):
      seq=Sequence()
      seq.id=record.id
      seq.sequence=record.seq
      sequences.append(seq)
  
  repeated_sequences=set()
  for seq in sequences:
      find_repeats(sequences, seq, repeated_sequences)
  
  print(repeated_sequences)
  
  def mySortFunc(e):
      return len(e.sequence)
  
  sequences.sort(key=mySortFunc)
  
  for seq in sequences:
      b= seq.id in repeated_sequences
      if not b:
          res+=">"+seq.id+"\n"
          res+=str(seq.sequence)+"\n"
  
  output_file = open(savefile, "w")
  
  output_file.write(res)
  
  output_file.close()
  
  ```
  
  EDIT: I see that's not exactly what OP is asking ...
  
  
  ---
  
  ### Answer by username_10 on 2021-01-28. Score: 0
  
  Your question isn't well formatted, for example list is used as a variable name where it should be reserved as a keyword...
  
  ```
  print (">"+list.group(1))
  
  ```
  
  However, the answer you're after is going to be something like:
  
  ```
  print (">"+set(name_of_list.group(1)))
  
  ```
  
  Anytime you want to get the unique elements of a list, just wrap it in set()
  
  
  ---
  
  