title: Does avg() on a dataset produce the most efficient RDD?
body: <p>As far as I understand, this is the most efficient way to calculate average in Spark: <a href="https://stackoverflow.com/a/40101724/1026271">Spark : Average of values instead of sum in reduceByKey using Scala</a>.</p>
  
  <p>My question is: if I use the high-level dataset with a groupby followed by Spark functions' avg(), will I get the same RDD under the hood? Can I trust Catalyst or I should use the low-level RDD? I mean, will writing low-level code yield better results than a dataset?</p>
  
  <p>Example code:</p>
  
  <pre><code>employees
    .groupBy($"employee")
    .agg(
      avg($"salary").as("avg_salary")
    )
  </code></pre>
  
  <p>Versus:</p>
  
  <pre><code>employees
  .mapValues(employee =&gt; (employee.salary, 1)) // map entry with a count of 1
  .reduceByKey {
    case ((sumL, countL), (sumR, countR)) =&gt; 
      (sumL + sumR, countL + countR)
  }
  .mapValues { 
    case (sum , count) =&gt; sum / count 
  }
  </code></pre>