Answers.Body: <p>The solution posted by @el_oso and @Corralien <a href="https://stackoverflow.com/questions/76221113/refactoring-pandas-using-an-iterator-via-chunksize">here</a> was as follows:</p>
  <blockquote>
  <p>if you're already having memory issues reading the file, the below will actually read and write to the new file in chunks.</p>
  </blockquote>
  <pre><code>cols_to_keep = {'col1', 'col2', 'col3'}
  df = pd.read_csv('myinfile.csv', chunksize=100, usecols=cols_to_keep)
  with open('my.csv', 'w') as f:
      f.write('\t'.join(cols_to_keep) + '\n')
      for chunk in df:
          #do your processing here, appending that chunk to the file
          chunk.to_csv(f, header=None, sep='\t') 
  </code></pre>
  <blockquote>
  <p>f is _io.TextIOWrapper i.e. a stream you can consistently write to in small memory chunks</p>
  </blockquote>
  <hr />
  <p>This is very elegant because there are two components which are both cool and highlight why <code>pandas</code> is so widely used.</p>
  <ol>
  <li>Instead of loading a large file and then dumping stuff thats not wanted: don't load junk at point source.</li>
  <li>Simply open the write file outside the chunking loop will then stream the chunks to straight to disk and once the final chunking loop completes the file snaps shut.</li>
  </ol>
  <p>The process is</p>
  <ul>
  <li><p>Load a chunk of data, but don't load any 'junk'.</p>
  </li>
  <li><p>Process the chunk of data and write it to disk.</p>
  </li>
  <li><p>Load another chunk of data repeating the above two steps.</p>
  </li>
  <li><p>Continue until the entire file is processed.</p>
  </li>
  </ul>
  <p>The &quot;do processing&quot; is an OO module (there's alot going on) and the chunk is actually <code>self.chunk</code>. None of the original code needs to change its simply restructured inside the <code>open</code> loop to stream chunks to disk.</p>
  <p>The beauty of this solution is <code>chunksize</code> is a variable, here <code>100</code> is just the example. Python's iterator called <code>yield</code> is one line at a time iteration, but this the automates a defined 'chunk' at a time. Thus the runtime speed can be tuned against the &quot;RAM bottleneck&quot;.</p>
  <p><strong>Testing</strong> The chunked outfile was then reduced via <code>groupby</code> to remove the &quot;duplicates&quot; and frequencies summed (code not shown). The <code>chunking</code> OO module output file is identical to &quot;non-chunked&quot; OO module and written back into the module.</p>
  <p>Its worth upvoting the SO answers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-15T20:26:29.150Z
Answers.LastActivityDate: 2023-05-15T20:32:24.040Z
Answers.LastEditDate: 2023-05-15T20:32:24.040Z
Answers.UserDisplayName: M__
Body: <p><em>This question was also asked on <a href="https://stackoverflow.com/questions/76221113/refactoring-pandas-using-an-iterator-via-chunksize">Stack Overflow</a></em></p>
  <p><strong>Bioinformatics rationale</strong> eggNOG files can be very big and sump all available RAM for regular to medium sized desktops.</p>
  <hr />
  <p>I am looking for advice on using a pandas iterator.</p>
  <p>I performed a parsing operation using Python pandas, the size of the input files is  eggNOG a program used in metagenomic gene function identification but has broader application. The files are large and parsing is resulting in 'RAM bottleneck' phenomenon and will not processing/parse the file.</p>
  <p>The obvious solution is to shift to an iterator, which for pandas is the  chunksize option</p>
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.read_csv('myinfile.csv', sep=&quot;\t&quot;, chunksize=100)
  </code></pre>
  <p>Whats changed with the original code is the <code>chunksize=100</code> bit, forcing an iterator.</p>
  <p>The next step is just to perform a simple operation, dropping a few columns and moving all '-' characters to np.nan then writing the whole file to disk.</p>
  <pre><code>df.drop(['score', 'evalue', 'Description', 'EC', 'PFAMs'],axis=1).replace('-', np.nan)
  df.to_csv('my.csv',sep='\t',index=False)
  </code></pre>
  <p>How is the step performed under an iterator?</p>
  
Tags: <python><phylogenetics><phylogeny><pandas><kegg>
Title: Refactoring pandas using an iterator via chunksize
ThreadText: # Refactoring pandas using an iterator via chunksize
  
  (Asked by: username_0 on 2023-05-15)
  
  *This question was also asked on [Stack Overflow](https://stackoverflow.com/questions/76221113/refactoring-pandas-using-an-iterator-via-chunksize)*
  
  **Bioinformatics rationale** eggNOG files can be very big and sump all available RAM for regular to medium sized desktops.
  
  ---
  
  I am looking for advice on using a pandas iterator.
  
  I performed a parsing operation using Python pandas, the size of the input files is eggNOG a program used in metagenomic gene function identification but has broader application. The files are large and parsing is resulting in 'RAM bottleneck' phenomenon and will not processing/parse the file.
  
  The obvious solution is to shift to an iterator, which for pandas is the chunksize option
  
  ```
  import pandas as pd
  import numpy as np
  
  df = pd.read_csv('myinfile.csv', sep="\t", chunksize=100)
  
  ```
  
  Whats changed with the original code is the `chunksize=100` bit, forcing an iterator.
  
  The next step is just to perform a simple operation, dropping a few columns and moving all '-' characters to np.nan then writing the whole file to disk.
  
  ```
  df.drop(['score', 'evalue', 'Description', 'EC', 'PFAMs'],axis=1).replace('-', np.nan)
  df.to_csv('my.csv',sep='\t',index=False)
  
  ```
  
  How is the step performed under an iterator?
  
  **Tags:** python, phylogenetics, phylogeny, pandas, kegg
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-05-15. Score: 0 | Accepted answer
  
  The solution posted by @el\_oso and @Corralien [here](https://stackoverflow.com/questions/76221113/refactoring-pandas-using-an-iterator-via-chunksize) was as follows:
  
  > if you're already having memory issues reading the file, the below will actually read and write to the new file in chunks.
  
  ```
  cols_to_keep = {'col1', 'col2', 'col3'}
  df = pd.read_csv('myinfile.csv', chunksize=100, usecols=cols_to_keep)
  with open('my.csv', 'w') as f:
      f.write('\t'.join(cols_to_keep) + '\n')
      for chunk in df:
          #do your processing here, appending that chunk to the file
          chunk.to_csv(f, header=None, sep='\t') 
  
  ```
  
  > f is \_io.TextIOWrapper i.e. a stream you can consistently write to in small memory chunks
  
  ---
  
  This is very elegant because there are two components which are both cool and highlight why `pandas` is so widely used.
  
  1. Instead of loading a large file and then dumping stuff thats not wanted: don't load junk at point source.
  2. Simply open the write file outside the chunking loop will then stream the chunks to straight to disk and once the final chunking loop completes the file snaps shut.
  
  The process is
  
  * Load a chunk of data, but don't load any 'junk'.
  * Process the chunk of data and write it to disk.
  * Load another chunk of data repeating the above two steps.
  * Continue until the entire file is processed.
  
  The "do processing" is an OO module (there's alot going on) and the chunk is actually `self.chunk`. None of the original code needs to change its simply restructured inside the `open` loop to stream chunks to disk.
  
  The beauty of this solution is `chunksize` is a variable, here `100` is just the example. Python's iterator called `yield` is one line at a time iteration, but this the automates a defined 'chunk' at a time. Thus the runtime speed can be tuned against the "RAM bottleneck".
  
  **Testing** The chunked outfile was then reduced via `groupby` to remove the "duplicates" and frequencies summed (code not shown). The `chunking` OO module output file is identical to "non-chunked" OO module and written back into the module.
  
  Its worth upvoting the SO answers.
  
  
  ---
  
  