title: Why is pandas.grouby.mean so much faster than paralleled implementation
body: <p>I was using the pandas grouby mean function like the following on a very large dataset: </p>
  
  <pre><code>import pandas as pd
  df=pd.read_csv("large_dataset.csv")
  df.groupby(['variable']).mean() 
  </code></pre>
  
  <p>It looks like the function is not using multi-processing, and therefore, I implemented a paralleled version: </p>
  
  <pre><code>import pandas as pd 
  from multiprocessing import Pool, cpu_count 
  
  def meanFunc(tmp_name, df_input): 
      df_res=df_input.mean().to_frame().transpose()
      return df_res 
  
  def applyParallel(dfGrouped, func):
      num_process=int(cpu_count())
      with Pool(num_process) as p: 
          ret_list=p.starmap(func, [[name, group] for name, group in dfGrouped])
      return pd.concat(ret_list)
  
  applyParallel(df.groupby(['variable']), meanFunc)
  </code></pre>
  
  <p>However, it seems that pandas implementation is still <strong>way</strong> faster than my parallel implementation. </p>
  
  <p>I am looking at the <a href="https://github.com/pandas-dev/pandas/blob/v0.24.1/pandas/core/groupby/groupby.py#L1082-L1138" rel="nofollow noreferrer">source code</a> for pandas groupby, and I see that it is using cython. Is that the reason? </p>
  
  <pre><code>def _cython_agg_general(self, how, alt=None, numeric_only=True,
                          min_count=-1):
      output = {}
      for name, obj in self._iterate_slices():
          is_numeric = is_numeric_dtype(obj.dtype)
          if numeric_only and not is_numeric:
              continue
  
          try:
              result, names = self.grouper.aggregate(obj.values, how,
                                                     min_count=min_count)
          except AssertionError as e:
              raise GroupByError(str(e))
          output[name] = self._try_cast(result, obj)
  
      if len(output) == 0:
          raise DataError('No numeric types to aggregate')
  
      return self._wrap_aggregated_output(output, names)
  </code></pre>