title: Median / quantiles within PySpark groupBy
body: <p>I would like to calculate group quantiles on a Spark dataframe (using PySpark). Either an approximate or exact result would be fine. I prefer a solution that I can use within the context of <code>groupBy</code> / <code>agg</code>, so that I can mix it with other PySpark aggregate functions. If this is not possible for some reason, a different approach would be fine as well.</p>
  
  <p><a href="https://stackoverflow.com/questions/31432843/how-to-find-median-and-quantiles-using-spark">This question</a> is related but does not indicate how to use <code>approxQuantile</code> as an aggregate function.</p>
  
  <p>I also have access to the <code>percentile_approx</code> Hive UDF but I don't know how to use it as an aggregate function.</p>
  
  <p>For the sake of specificity, suppose I have the following dataframe:</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark import SparkContext
  import pyspark.sql.functions as f
  
  sc = SparkContext()    
  
  df = sc.parallelize([
      ['A', 1],
      ['A', 2],
      ['A', 3],
      ['B', 4],
      ['B', 5],
      ['B', 6],
  ]).toDF(('grp', 'val'))
  
  df_grp = df.groupBy('grp').agg(f.magic_percentile('val', 0.5).alias('med_val'))
  df_grp.show()
  </code></pre>
  
  <p>Expected result is:</p>
  
  <pre><code>+----+-------+
  | grp|med_val|
  +----+-------+
  |   A|      2|
  |   B|      5|
  +----+-------+
  </code></pre>