Answers.Body: <p>You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.</p>
  <p>The solution is to either:</p>
  <blockquote>
  <ol>
  <li><p>find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.</p>
  </li>
  <li><p>parallelise the calculation (an excellent way to overcome RAM problems), or,</p>
  </li>
  <li><p>find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.</p>
  </li>
  </ol>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-23T22:56:35.087Z
Answers.LastActivityDate: 2018-12-23T22:56:35.087Z
Answers.LastEditDate: 2020-06-18T08:30:41.743Z
Answers.UserDisplayName: M__
Body: <p>I have <a href="https://github.com/tbenavi1/smudgeplot/blob/master/hetkmers.py" rel="nofollow noreferrer">a script</a> that extracts the set of unique kmer pairs from a kmer dump file (&amp; then using them downstream to guess ploidy).</p>
  
  <p>The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.</p>
  
  <p>I don't really get it; the script actually prints the last line </p>
  
  <blockquote>
    <p>*_families_2.tsv and *_coverages_2.tsv files saved.</p>
  </blockquote>
  
  <p>, but then it runs further (&amp; forever). I thought it could be related to longer time to free the memory as discussed <a href="https://stackoverflow.com/questions/2412715/python-program-doesnt-quit-when-finished">here</a>, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).</p>
  
  <p>Does anybody an idea where the problem could be? Or how to figure out?</p>
  
  <p>The reported problem on <a href="https://github.com/tbenavi1/smudgeplot/issues/18" rel="nofollow noreferrer">GitHub</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T15:01:21.390Z
Comments.Text: Can you trace the memory usage of the progress to see if it's related to freeing memory? we had a similar issue some time ago, tried a lot of things like disabling garbage collection etc. but never resolved it. Also have you tried running on another system? our issue was linked to a particular system with an old OS/software
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T15:02:10.457Z
Comments.Text: Also this might fit better on Stack Overflow (although they might complain about it being no small and reproducible)
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T15:02:31.480Z
Comments.Text: After the last line is printed the program seems to occupy exactly the same amount of memory over time.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T15:04:05.343Z
Comments.Text: I was thinking about posting it to Stack Overflow, but since it's kmer related problem and since I prefer to interact with this community I have decided to post it here.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T15:33:17.710Z
Comments.Text: As the author [suggested](https://github.com/tbenavi1/smudgeplot/issues/18#issuecomment-426609963), this is probably caused by a deadlock in multi-threading.
Comments.UserDisplayName: user172818
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-24T02:07:17.993Z
Comments.Text: you probably need to do "from multiprocessing import Pool" and not "Queue" or "spawn"
Comments.UserDisplayName: user3479780
Tags: <python><k-mer>
Title: python does not quit when the input file is too big
ThreadText: # python does not quit when the input file is too big
  
  (Asked by: username_0 on 2018-12-20)
  
  I have [a script](https://github.com/tbenavi1/smudgeplot/blob/master/hetkmers.py) that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).
  
  The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.
  
  I don't really get it; the script actually prints the last line
  
  > \*\_families\_2.tsv and \*\_coverages\_2.tsv files saved.
  
  , but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed [here](https://stackoverflow.com/questions/2412715/python-program-doesnt-quit-when-finished), but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).
  
  Does anybody an idea where the problem could be? Or how to figure out?
  
  The reported problem on [GitHub](https://github.com/tbenavi1/smudgeplot/issues/18).
  
  **Tags:** python, k-mer
  
  > **Comments:**
  > **username_1**: Can you trace the memory usage of the progress to see if it's related to freeing memory? we had a similar issue some time ago, tried a lot of things like disabling garbage collection etc. but never resolved it. Also have you tried running on another system? our issue was linked to a particular system with an old OS/software  
  > **username_1**: Also this might fit better on Stack Overflow (although they might complain about it being no small and reproducible)  
  > **username_0**: After the last line is printed the program seems to occupy exactly the same amount of memory over time.  
  > **username_0**: I was thinking about posting it to Stack Overflow, but since it's kmer related problem and since I prefer to interact with this community I have decided to post it here.  
  > **username_2**: As the author [suggested](https://github.com/tbenavi1/smudgeplot/issues/18#issuecomment-426609963), this is probably caused by a deadlock in multi-threading.  
  > **username_3**: you probably need to do "from multiprocessing import Pool" and not "Queue" or "spawn"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2018-12-23. Score: 2
  
  You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.
  
  The solution is to either:
  
  > 1. find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.
  > 2. parallelise the calculation (an excellent way to overcome RAM problems), or,
  > 3. find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.
  
  
  ---
  
  