title: Pyspark - Calculate Median Absolute Percent Error with groupby
body: <p>I'm able to calculate the Median Absolute Error with this function:</p>
  
  <pre><code>from pyspark.sql import Window
  
  def compute_Median_Abs_Err(df, expected_col, actual_col):
  
      grp_window = Window.partitionBy('grp')
  
      magic_percentile = F.expr('percentile_approx(abserror, 0.5)')
  
      med_abs_err = df.withColumn("abserror",
                             f.abs(f.col(actual_col) - f.col(expected_col)
                             )).groupby('start_month', 'start_dt'
                                       ).agg(magic_percentile.alias("med_abs_error")
                                            )
  
      return(med_abs_err)
  </code></pre>
  
  <p>Which can be calculate with this equation: </p>
  
  <p>﻿<code>MEDIAN(abs(predictions - actuals))</code></p>
  
  <p>I'd like to be able to calculate the Median Absolute Percent Error, calculated with this equation: </p>
  
  <p><code>MEDIAN(﻿ abs(predictions - actuals) / actuals )</code></p>
  
  <p>I thought I had it correctly with this:</p>
  
  <pre><code>from pyspark.sql import Window
  
  def compute_Median_Perc_Err(df, expected_col, actual_col):
  
      grp_window = Window.partitionBy('grp')
  
      magic_percentile = f.expr('percentile_approx(abserror, 0.5)')
  
      med_perc_err = df.withColumn("abserror",
                             f.abs(f.col(actual_col) - f.col(expected_col)
                             )).groupby('start_month', 'start_dt'
                                       ).agg(magic_percentile.alias("med_abs_error"), f.avg(f.col(actual_col)).alias("mean")
                                            ).withColumn("med_perc_error", f.col("med_abs_error") / f.col("mean"))
  
  
      return(med_perc_err)
  </code></pre>
  
  <p>But I realized with this, I am not dividing by the <code>actuals</code> before taking the <code>median</code>. I should divide by the actuals first, then take the median of that column.</p>
  
  <p>How do I write this code snippet to divide by the actuals first, since I still need to take <code>.agg(f.avg(f.col("actuals"))</code> after the groupby to get an accurate mean?</p>