title: Simple Parallel processing chunks of pandas Dataframe with groupby and aggregate operation
body: <p>Edit: I found the mistake, i forgot to actually return the data in my aggregation function... whoops.
  Anyway, there's a couple of methods listed below, i'll leave them here for reference.</p>
  
  <p>Original:
  I need to apply a rather computationally expensive groupby and aggregate on a pandas dataframe.</p>
  
  <pre><code>df.groupby(['Col1','Col2'],as_index=False).aggregate(lambda x: list(x))
  </code></pre>
  
  <p>I cannot use Map for this (or maybe i do, i have no idea anymore), since i need access to multiple lines of the data in order for the two column groupby to work.<br>
  Using Pool.map just returns the same df, as is expected, because it passes the data row by row instead of group-by-group or at least as a whole chunk.</p>
  
  <p>What i want is simple: </p>
  
  <ul>
  <li>split the Dataframe in n Slices/Chunks (np.array_split works well for this i think),</li>
  <li>apply the groupby and aggregate on each slice (just a normal operation on each slice)</li>
  <li>then merge the slices together (pd.concat) </li>
  </ul>
  
  <p>this really shouldn't be very hard should it? But every example and tutorial is fixated on row-by-row operations which make the groupby/aggregate do nothing.</p>
  
  <p>Thanks for you help :)</p>
  
  <p>EDIT:</p>
  
  <p>I tried:</p>
  
  <pre><code>from multiprocessing import Pool
  num_partitions = 10 #number of partitions to split dataframe
  num_cores = 4 #number of cores on your machine
  def parallelize_dataframe(df, func):
      df_split = np.array_split(df, num_partitions)
      pool = Pool(num_cores)
      df = pd.concat(pool.map(func, [group for name,group in df.groupby(['Col1','Col2'])]))
      pool.close()
      pool.join()
      return df
  def aggregate_fun(data):
      data = data.groupby(['Col1','Col2'],as_index=False).aggregate(lambda x: list(x))
      return data
  </code></pre>
  
  <p>EDIT2:</p>
  
  <pre><code>from concurrent.futures import ProcessPoolExecutor, as_completed
  ppe = ProcessPoolExecutor(4)
  
  futures = []
  results = []
  for group in np.split(df, 4):
      p = ppe.submit(aggregate_fun, group)
      futures.append(p)
  
  for future in as_completed(futures):
      r = future.result()
      results.append(r)
  
  df_output = pd.concat(results) 
  </code></pre>
  
  <p>Both methods works, so long as you don't forget to have your function actually return something :P 
  It is however not as big a speedup as i would hope, 
  note that i forgot to allocate the data variable before returning or just returning the aggregation result... Goes to show that coding should not be done late in the afternoon.</p>