title: Python multiprocessing with generator
body: <p>I'm trying to process a file(every line is a json document). The size of the file can go up to 100's of mbs to gb's. So I wrote a generator code to fetch each document line by line from file.</p>
  
  <pre><code>def jl_file_iterator(file):
      with codecs.open(file, 'r', 'utf-8') as f:
          for line in f:
              document = json.loads(line)
              yield document
  </code></pre>
  
  <p>My system has 4 cores, So I would like to process 4 lines of the file in parallel. Currently I have this code which takes 4 lines at a time and calls the code for parallel processing</p>
  
  <pre><code>threads = 4
  files, i = [], 1
  for jl in jl_file_iterator(input_path):
      files.append(jl)
      if i % (threads) == 0:
          # pool.map(processFile, files)
          parallelProcess(files, o)
          files = []
      i += 1
  
  if files:
      parallelProcess(files, o)
      files = []
  </code></pre>
  
  <p>This is my code where actual processing happens</p>
  
  <pre><code>def parallelProcess(files, outfile):
      processes = []
      for i in range(len(files)):
          p = Process(target=processFile, args=(files[i],))
          processes.append(p)
          p.start()
      for i in range(len(files)):
          processes[i].join()
  
  def processFile(doc):
      extractors = {}
      ... do some processing on doc
      o.write(json.dumps(doc) + '\n')
  </code></pre>
  
  <p>As you can see I wait for all the 4 lines to finish processing before I send the next 4 files to process. But what I would like to do is as soon as one process finish processing file I want to start the next line to be assigned to realeased processor. How do I do that? </p>
  
  <p>PS: The problem is since its an generator I cannot load all the files and use something like map to run the processes. </p>
  
  <p>Thanks for your help</p>