title: Multiprocessing - asynchronous processes, but block based on function argument
body: <p>I am using multiprocessing to call an external executable over and over with many different arguments. For me the executable is a black box and reads/writes to files in its directory. Therefore, I have 4 copies of the executable, each in their own directory. This allows me to run the executable 4 at a time without collision.</p>
  
  <p>Originally I tried something like this</p>
  
  <pre><code>PROCESSES = 4
  with multiprocessing.Pool(PROCESSES) as pool:
      exe_list = [ii for ii in range(4)]*10
      TASKS = ((pyfcn, (ii, ex)) for ii, ex in enumerate(exe_list))
  
      results = [pool.apply_async(calculate, t) for t in TASKS]
  
      print('Ordered results using pool.apply_async():')
      for r in results:
          print('\t', r.get())
      print()
  </code></pre>
  
  <p>which demonstrates calling a python function which then calls the external executable. <code>exe_list</code> is a list that has <code>[0,1,2,3,0,1,2,...]</code> specifying the executables. <code>ii</code> is just a stand in for the real function arguments that also get passed to the executable. Although this method returns the results in order (which I want), if for example the process using executable 2 isn't finished before 3, 0, and 1 are finished, then I get a collision when another process tries to begin using executable 2.</p>
  
  <p>How can I get the results in order, use the processors asynchronously, but also prevent collisions of the external executable that is being called? Basically I want to "block" only if I've gone through the cycle of 4 executables and one isn't finished.</p>
  
  <p>Maybe 7/10 executable calls finish quickly enough, but it's inevitable that I get a collision because of the fact that some of the function arguments take longer for the executable to process.</p>