Answers.Body: <p>I wrote the answer <a href="https://bioinformatics.stackexchange.com/a/13710/104">here</a> and while I'm not clear on all aspects of this question, I can offer two suggestions to reduce the memory footprint considerably while maintaining high speed.</p>
  <p>First, rather than appending the records to a list you could use a generator that generates elements on-the-fly rather than storing them in memory, e.g.</p>
  <pre><code>seen = set()
  
  def yield_seqs():
      for record in SeqIO.parse(&quot;b4r2.fasta&quot;, &quot;fasta&quot;):  
          if record.seq not in seen:
              seen.add(record.seq)
              yield record
  
  SeqIO.write(yield_seqs(), &quot;output.fasta&quot;, &quot;fasta&quot;)
  </code></pre>
  <p>Second you could save even more memory, <strong>but with the risk of false positives</strong>, by using a <a href="https://en.wikipedia.org/wiki/Bloom_filter" rel="nofollow noreferrer">bloom filter</a>, e.g. using <a href="https://github.com/hiway/python-bloom-filter" rel="nofollow noreferrer">python-bloom-filter</a>, just replace the set with <code>seen = BloomFilter(max_elements=10000, error_rate=0.001)</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-07T19:30:14.027Z
Answers.Text: thanks for the generator hint.
Answers.UserDisplayName: pippo1980
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-07T18:43:28.810Z
Answers.LastActivityDate: 2021-03-07T18:43:28.810Z
Answers.UserDisplayName: Chris_Rands
Answers.Body: <p>FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:</p>
  <pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  ......
  
  SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
              
  record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)
  
  </code></pre>
  <p>the line:</p>
  <pre><code> if record_iter.seq != record_output.seq  :
  
  </code></pre>
  <p>will throw:</p>
  <pre><code>
  AttributeError: 'list' object has no attribute 'seq'
  
  </code></pre>
  <p>After spending some timde on it I realized that:</p>
  <pre><code>print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
  </code></pre>
  <p>is returning:</p>
  <pre><code>&lt;class 'list'&gt; 
  </code></pre>
  <p>but jus using :</p>
  <pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch[0] 
  </code></pre>
  <p>now :</p>
  <pre><code>print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
  </code></pre>
  <p>is returning:</p>
  <pre><code>&lt;class 'Bio.SeqRecord.SeqRecord'&gt; 
  </code></pre>
  <p>no need to save and read the Bio.SeqRecord.SeqRecord' anymore</p>
  <p>so this is just half the question answered so no need to save and read the record on file</p>
  <p>but even if speed increases the memory usage goes up :</p>
  <pre><code>data_input_iter :  48 
  
  Run time is 1.442115306854248
  Maximum memory usage: 110.09765625
  <span class="math-container">```</span>
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T14:09:06.083Z
Answers.LastActivityDate: 2021-11-02T22:19:03.440Z
Answers.LastEditDate: 2021-11-02T22:19:03.440Z
Answers.UserDisplayName: pippo1980
Body: <p>I need help with my Biopython code, I am kind of new to Python and Biopython so please don't be harsh on me.</p>
  <p>Everything started from here: <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a></p>
  <p>I answered copying code from here: <a href="https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python">Remove Redundant Sequences from FASTA file in Python</a> but I am still bugged about the OP's first question: <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a></p>
  <p>So I decided to try to minimize the code footprint on RAM. I ended up copying from Biopython documentation <a href="https://biopython.org/wiki/Split_large_file" rel="nofollow noreferrer">Split large file</a> and producing:</p>
  <pre><code>#!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  &quot;&quot;&quot;
  Created on Thu Nov  11 17:40:03 2020
  
  @author: Pietro
  
  
  &quot;&quot;&quot;
  
  from memory_profiler import memory_usage
  
  import memory_profiler 
  
  
  def batch_iterator(iterator, batch_size):
      &quot;&quot;&quot; from https://biopython.org/wiki/Split_large_file
      &quot;&quot;&quot;
      entry = True  # Make sure we loop once
      while entry:
          batch = []
          while len(batch) &lt; batch_size:
              try:
                  entry = iterator.__next__()
              except StopIteration:
                  entry = None
              if entry is None:
                  # End of file
                  break
              batch.append(entry)
          if batch:
              yield batch
  
  
  # @profile
  def f():
      
      from Bio import SeqIO 
      
      import time
      
      import sys
     
      start = time.time() 
  
      data_input_iter = SeqIO.parse(&quot;Newtextfile-03&quot;, &quot;fasta&quot;)
      print('data_input_iter : ',sys.getsizeof(data_input_iter),'\n')
      
  
      c=0
      for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  
          val = True
          c += 1
          if c == 1:
              with open(&quot;removed-double_Newtextfile-03&quot;, 'w+') as output_file:
                      SeqIO.write(record_iter, output_file, &quot;fasta&quot;)
                      # print('copy   : ',(record_iter))
                      output_file.flush()
              output_file.close()
          
          else:
              
              SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
              
              record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)
              
              data_output_iter = SeqIO.parse(&quot;removed-double_Newtextfile-03&quot;, &quot;fasta&quot;)
              
              d = 0
              
              try:
                  for i , batch in enumerate(batch_iterator(data_output_iter  , 1)):
                          SeqIO.write(batch, &quot;temp_file2&quot;, &quot;fasta&quot;)
                      
                          record_output = SeqIO.read(&quot;temp_file2&quot;, &quot;fasta&quot;)
                      
                          d += 1
                          if record_iter.seq != record_output.seq  :
                              # print('pass   : ',record_iter.name)
                              pass
                          else:
                              val = False
                              # print('false')
                              break
  
                  
              except:
                  print('ERROR !!!!!!!!!!!')
                  quit()
  
              if val:
                          # print('copy   : ',record_iter.name)
                          with open(&quot;removed-double_Newtextfile-03&quot;, 'a+') as output_file:
                              output_file.write(str(record_iter.format(&quot;fasta&quot;)))
                          output_file.close()
              del data_output_iter
      
      end = time.time()
      
      print(f&quot;Run time is {(end- start)}&quot;) 
  
  
  mem_usage = memory_usage(f)
  # print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
  print('Maximum memory usage: %s' % max(mem_usage))
  
  
  # f()
  
  
  </code></pre>
  <p>but I have to major problems :</p>
  <p>FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:</p>
  <pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  ......
  
  SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
              
  record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)
  
  </code></pre>
  <p>the line:</p>
  <pre><code> if record_iter.seq != record_output.seq  :
  
  </code></pre>
  <p>will throw:</p>
  <pre><code>AttributeError: 'list' object has no attribute 'format'
  
  AttributeError: 'list' object has no attribute 'seq'
  
  </code></pre>
  <p>you get first error and the if remove format from:</p>
  <pre><code>output_file.write(str(record_iter.format(&quot;fasta&quot;)))
  </code></pre>
  <p>and #quit() you get the second error
  How should I cope with this ? I mean is there a way to bypass the save and read lines ?</p>
  <p>SECOND : and more important. I am sure my code doesn't use a set and a list .... and that's why is extremely slow compared to the copied <a href="https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python">Remove Redundant Sequences from FASTA file in Python</a> posted below too. But how does it use even more RAM that the fastest code ? I tried to minimize the memory footprint but wasn't able !! Am I using a bad way to try to calculate the amount of RAM used ? And does Biopythom SeqIO.parse object already is the optimal choice for , lets say &quot;2/4 Gb size&quot; fasta files on a laptop with just 4Gb of RAM (remember the original OP post  <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a>). Maybe my test file is not big enough ? Does anybody have a 2GB fasta file with a dozen of duplicates in it ?</p>
  <p><a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a> memory usage:</p>
  <pre><code>size of parser :  48
  memory seen :  32984
  memory records :  2888
  Run time is 0.0008047382036844889
  Maximum memory usage: 55.359375
  </code></pre>
  <p>my code memory usage and speed:</p>
  <pre><code>data_input_iter :  48 
  
  Run time is 37.781240463256836
  Maximum memory usage: 56.40234375
  </code></pre>
  <p>My fasta test set is: <a href="https://easyupload.io/z2yss1" rel="nofollow noreferrer">https://easyupload.io/z2yss1</a></p>
  <p>Biopython method to remove duplicated from  <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a>:</p>
  <pre><code>#!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  &quot;&quot;&quot;
  copied from https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python
  
  uses set() instead of list for seen
  
  should work faster for big files
  
  &quot;&quot;&quot;
  
  from memory_profiler import memory_usage
  
  def f():
      
      import sys
  
      from Bio import SeqIO
      import time
      
      start = time.time() 
      
      seen = set()
      records = []
      
      
      parser = SeqIO.parse(&quot;Newtextfile-03&quot;, &quot;fasta&quot;)
      for record in parser:  
          # print('running')
          if str(record.seq) not in seen:
              seen.add(str(record.seq))
              records.append(record)
              # print('copying')
      
      
      print('size of parser : ',sys.getsizeof(parser) )
      print('memory seen : ',sys.getsizeof(seen))
      print('memory records : ',sys.getsizeof(records))
      
      #writing to a fasta file
      SeqIO.write(records, &quot;removed-double_Newtextfile-03-from-002&quot;, &quot;fasta&quot;)
      end = time.time()
      
      print(f&quot;Run time is {(end- start)/60}&quot;) 
  
  mem_usage = memory_usage(f)
  # print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
  print('Maximum memory usage: %s' % max(mem_usage))
  
  # f()
  </code></pre>
  
Tags: <python><fasta><seqio><biopython>
Title: Remove Redundant Sequences from FASTA file with Biopython reducing memory footprint
ThreadText: # Remove Redundant Sequences from FASTA file with Biopython reducing memory footprint
  
  (Asked by: username_0 on 2021-03-07)
  
  I need help with my Biopython code, I am kind of new to Python and Biopython so please don't be harsh on me.
  
  Everything started from here: [Remove duplicated sequences in FASTA with Python](https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550)
  
  I answered copying code from here: [Remove Redundant Sequences from FASTA file in Python](https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python) but I am still bugged about the OP's first question: [Remove duplicated sequences in FASTA with Python](https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550)
  
  So I decided to try to minimize the code footprint on RAM. I ended up copying from Biopython documentation [Split large file](https://biopython.org/wiki/Split_large_file) and producing:
  
  ```
  #!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  """
  Created on Thu Nov  11 17:40:03 2020
  
  @author: Pietro
  
  """
  
  from memory_profiler import memory_usage
  
  import memory_profiler 
  
  def batch_iterator(iterator, batch_size):
      """ from https://biopython.org/wiki/Split_large_file
      """
      entry = True  # Make sure we loop once
      while entry:
          batch = []
          while len(batch) < batch_size:
              try:
                  entry = iterator.__next__()
              except StopIteration:
                  entry = None
              if entry is None:
                  # End of file
                  break
              batch.append(entry)
          if batch:
              yield batch
  
  # @profile
  def f():
      
      from Bio import SeqIO 
      
      import time
      
      import sys
     
      start = time.time() 
  
      data_input_iter = SeqIO.parse("Newtextfile-03", "fasta")
      print('data_input_iter : ',sys.getsizeof(data_input_iter),'\n')
      
  
      c=0
      for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  
          val = True
          c += 1
          if c == 1:
              with open("removed-double_Newtextfile-03", 'w+') as output_file:
                      SeqIO.write(record_iter, output_file, "fasta")
                      # print('copy   : ',(record_iter))
                      output_file.flush()
              output_file.close()
          
          else:
              
              SeqIO.write(record_iter, "temp_file", "fasta")
              
              record_iter = SeqIO.read("temp_file", "fasta")
              
              data_output_iter = SeqIO.parse("removed-double_Newtextfile-03", "fasta")
              
              d = 0
              
              try:
                  for i , batch in enumerate(batch_iterator(data_output_iter  , 1)):
                          SeqIO.write(batch, "temp_file2", "fasta")
                      
                          record_output = SeqIO.read("temp_file2", "fasta")
                      
                          d += 1
                          if record_iter.seq != record_output.seq  :
                              # print('pass   : ',record_iter.name)
                              pass
                          else:
                              val = False
                              # print('false')
                              break
  
                  
              except:
                  print('ERROR !!!!!!!!!!!')
                  quit()
  
              if val:
                          # print('copy   : ',record_iter.name)
                          with open("removed-double_Newtextfile-03", 'a+') as output_file:
                              output_file.write(str(record_iter.format("fasta")))
                          output_file.close()
              del data_output_iter
      
      end = time.time()
      
      print(f"Run time is {(end- start)}") 
  
  mem_usage = memory_usage(f)
  # print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
  print('Maximum memory usage: %s' % max(mem_usage))
  
  # f()
  
  ```
  
  but I have to major problems :
  
  FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:
  
  ```
  for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  ......
  
  SeqIO.write(record_iter, "temp_file", "fasta")
              
  record_iter = SeqIO.read("temp_file", "fasta")
  
  ```
  
  the line:
  
  ```
   if record_iter.seq != record_output.seq  :
  
  ```
  
  will throw:
  
  ```
  AttributeError: 'list' object has no attribute 'format'
  
  AttributeError: 'list' object has no attribute 'seq'
  
  ```
  
  you get first error and the if remove format from:
  
  ```
  output_file.write(str(record_iter.format("fasta")))
  
  ```
  
  and #quit() you get the second error
  How should I cope with this ? I mean is there a way to bypass the save and read lines ?
  
  SECOND : and more important. I am sure my code doesn't use a set and a list .... and that's why is extremely slow compared to the copied [Remove Redundant Sequences from FASTA file in Python](https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python) posted below too. But how does it use even more RAM that the fastest code ? I tried to minimize the memory footprint but wasn't able !! Am I using a bad way to try to calculate the amount of RAM used ? And does Biopythom SeqIO.parse object already is the optimal choice for , lets say "2/4 Gb size" fasta files on a laptop with just 4Gb of RAM (remember the original OP post [Remove duplicated sequences in FASTA with Python](https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550)). Maybe my test file is not big enough ? Does anybody have a 2GB fasta file with a dozen of duplicates in it ?
  
  [Remove duplicated sequences in FASTA with Python](https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550) memory usage:
  
  ```
  size of parser :  48
  memory seen :  32984
  memory records :  2888
  Run time is 0.0008047382036844889
  Maximum memory usage: 55.359375
  
  ```
  
  my code memory usage and speed:
  
  ```
  data_input_iter :  48 
  
  Run time is 37.781240463256836
  Maximum memory usage: 56.40234375
  
  ```
  
  My fasta test set is: <https://easyupload.io/z2yss1>
  
  Biopython method to remove duplicated from [Remove duplicated sequences in FASTA with Python](https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550):
  
  ```
  #!/usr/bin/env python3
  # -*- coding: utf-8 -*-
  """
  copied from https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python
  
  uses set() instead of list for seen
  
  should work faster for big files
  
  """
  
  from memory_profiler import memory_usage
  
  def f():
      
      import sys
  
      from Bio import SeqIO
      import time
      
      start = time.time() 
      
      seen = set()
      records = []
      
      
      parser = SeqIO.parse("Newtextfile-03", "fasta")
      for record in parser:  
          # print('running')
          if str(record.seq) not in seen:
              seen.add(str(record.seq))
              records.append(record)
              # print('copying')
      
      
      print('size of parser : ',sys.getsizeof(parser) )
      print('memory seen : ',sys.getsizeof(seen))
      print('memory records : ',sys.getsizeof(records))
      
      #writing to a fasta file
      SeqIO.write(records, "removed-double_Newtextfile-03-from-002", "fasta")
      end = time.time()
      
      print(f"Run time is {(end- start)/60}") 
  
  mem_usage = memory_usage(f)
  # print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
  print('Maximum memory usage: %s' % max(mem_usage))
  
  # f()
  
  ```
  
  **Tags:** python, fasta, seqio, biopython
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-07. Score: 2
  
  I wrote the answer [here](https://bioinformatics.stackexchange.com/a/13710/104) and while I'm not clear on all aspects of this question, I can offer two suggestions to reduce the memory footprint considerably while maintaining high speed.
  
  First, rather than appending the records to a list you could use a generator that generates elements on-the-fly rather than storing them in memory, e.g.
  
  ```
  seen = set()
  
  def yield_seqs():
      for record in SeqIO.parse("b4r2.fasta", "fasta"):  
          if record.seq not in seen:
              seen.add(record.seq)
              yield record
  
  SeqIO.write(yield_seqs(), "output.fasta", "fasta")
  
  ```
  
  Second you could save even more memory, **but with the risk of false positives**, by using a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), e.g. using [python-bloom-filter](https://github.com/hiway/python-bloom-filter), just replace the set with `seen = BloomFilter(max_elements=10000, error_rate=0.001)`
  
  > **Comments:**
  > **username_0**: thanks for the generator hint.  
  
  ---
  
  ### Answer by username_0 on 2021-04-06. Score: 0
  
  FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:
  
  ```
  for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch 
  ......
  
  SeqIO.write(record_iter, "temp_file", "fasta")
              
  record_iter = SeqIO.read("temp_file", "fasta")
  
  ```
  
  the line:
  
  ```
   if record_iter.seq != record_output.seq  :
  
  ```
  
  will throw:
  
  ```
  
  AttributeError: 'list' object has no attribute 'seq'
  
  ```
  
  After spending some timde on it I realized that:
  
  ```
  print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
  
  ```
  
  is returning:
  
  ```
  <class 'list'> 
  
  ```
  
  but jus using :
  
  ```
  for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
          record_iter = batch[0] 
  
  ```
  
  now :
  
  ```
  print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
  
  ```
  
  is returning:
  
  ```
  <class 'Bio.SeqRecord.SeqRecord'> 
  
  ```
  
  no need to save and read the Bio.SeqRecord.SeqRecord' anymore
  
  so this is just half the question answered so no need to save and read the record on file
  
  but even if speed increases the memory usage goes up :
  
  ```
  data_input_iter :  48 
  
  Run time is 1.442115306854248
  Maximum memory usage: 110.09765625
  ```
  
  ```
  
  
  ---
  
  