Answers.Body: <p>You've overloaded the memory. One route is to deploy &quot;chunking&quot;. pandas will do this. This is complicated to implement code wise however.</p>
  <p>My suggested solution (because it simpler) would be to do this via <code>pandas2</code> which uses Apache Arrow via PyArrow. You need to specify which method is being used at the onset of declaring the dataframe. Apache Arrow is for lifting large amounts of data. If this works there's no complicated coding you just <code>import pandas2 as pd</code> and in the dataframe construction describe the precise heavy lift method (I think there's an additional approach besides PyArrow).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T22:01:24.137Z
Answers.Text: Thank you! I was able to use a chunking method to calculate all the distances! Now the issue I'm having is I can't open the file to see the distances. Normally, I'd convert the .tsv file to a .meg file by going into the file and adding the proper headers to visualize the tree with MEGAX. But since I can't open the file I can't normally convert it. Any tips?
Answers.UserDisplayName: rimo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-11T01:02:32.987Z
Answers.Text: Hi @rimo, great! Can you put this as a separate question please? It should be easily solvable. If you do I'll answer it on Saturday, because I've work commitments.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-12T16:42:04.370Z
Answers.Text: Thanks @M__!! I appreciate your help! I just posted another question. Take a look when you can!
Answers.UserDisplayName: rimo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T21:04:33.840Z
Answers.LastActivityDate: 2024-01-02T21:10:01.900Z
Answers.LastEditDate: 2024-01-02T21:10:01.900Z
Answers.UserDisplayName: M__
Body: <p>I have a large tsv file (~15 GB) that is a kmer count matrix and I am trying to calculate jaccard distance of the counts but every time I try to run the following code the process ends before it's done with a <code>killed</code> error message in the STDOUT:</p>
  <pre class="lang-py prettyprint-override"><code>jaccard = pd.DataFrame(squareform(pdist(kmer_matrix, 'jaccard')))
  jaccard.to_csv(matrix_path+kmer+&quot;_&quot;+method+&quot;_jacDistance_noFilter.tsv&quot;, sep='\t')
  </code></pre>
  <p>This process works really well with smaller tsv files so I know the code works but I wanted to know if anyone knew what would help run the process quicker/completely. I know the problem must be with my RAM/CPU but is there a way to maybe break up the matrix to calculate everything in batches to make it easier to work with?</p>
  
Tags: <k-mer><errors><distance-matrix><troubleshooting>
Title: Process gets killed before completing
ThreadText: # Process gets killed before completing
  
  (Asked by: username_0 on 2024-01-02)
  
  I have a large tsv file (~15 GB) that is a kmer count matrix and I am trying to calculate jaccard distance of the counts but every time I try to run the following code the process ends before it's done with a `killed` error message in the STDOUT:
  
  ```py
  jaccard = pd.DataFrame(squareform(pdist(kmer_matrix, 'jaccard')))
  jaccard.to_csv(matrix_path+kmer+"_"+method+"_jacDistance_noFilter.tsv", sep='\t')
  
  ```
  
  This process works really well with smaller tsv files so I know the code works but I wanted to know if anyone knew what would help run the process quicker/completely. I know the problem must be with my RAM/CPU but is there a way to maybe break up the matrix to calculate everything in batches to make it easier to work with?
  
  **Tags:** k-mer, errors, distance-matrix, troubleshooting
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-02. Score: 3 | Accepted answer
  
  You've overloaded the memory. One route is to deploy "chunking". pandas will do this. This is complicated to implement code wise however.
  
  My suggested solution (because it simpler) would be to do this via `pandas2` which uses Apache Arrow via PyArrow. You need to specify which method is being used at the onset of declaring the dataframe. Apache Arrow is for lifting large amounts of data. If this works there's no complicated coding you just `import pandas2 as pd` and in the dataframe construction describe the precise heavy lift method (I think there's an additional approach besides PyArrow).
  
  > **Comments:**
  > **username_0**: Thank you! I was able to use a chunking method to calculate all the distances! Now the issue I'm having is I can't open the file to see the distances. Normally, I'd convert the .tsv file to a .meg file by going into the file and adding the proper headers to visualize the tree with MEGAX. But since I can't open the file I can't normally convert it. Any tips?  
  > **username_1**: Hi @username_0, great! Can you put this as a separate question please? It should be easily solvable. If you do I'll answer it on Saturday, because I've work commitments.  
  > **username_0**: Thanks @username_1!! I appreciate your help! I just posted another question. Take a look when you can!  
  
  ---
  
  