title: Python's multiprocessing and memory
body: <p>I am using <code>multiprocessing.imap_unordered</code> to perform a computation on a list of values:</p>
  
  <pre><code>def process_parallel(fnc, some_list):
      pool = multiprocessing.Pool()
      for result in pool.imap_unordered(fnc, some_list):
          for x in result:
              yield x
      pool.terminate()
  </code></pre>
  
  <p>Each call to <code>fnc</code> returns a HUGE object as a result, by design. I can store N instances of such object in RAM, where N ~ cpu_count, but not much more (not hundreds).</p>
  
  <p>Now, using this function takes up too much memory. The memory is entirely spent in the main process, not in the workers.</p>
  
  <p>How does <code>imap_unordered</code> store the finished results? I mean the results that were already returned by workers but not yet passed on to user. I thought it was smart and only computed them "lazily" as needed, but apparently not. </p>
  
  <p>It looks like since I cannot consume the results of <code>process_parallel</code> fast enough, the pool keeps queueing these huge objects from <code>fnc</code> somewhere, internally, and then blows up. Is there a way to avoid this? Limit its internal queue somehow?</p>
  
  <hr>
  
  <p>I'm using Python2.7. Cheers.</p>