title: Why is my Spark DataFrame much slower than RDD?
body: <p>I have a very simple Spark DataFrame, and when running a DataFrame groupby, the performance is terrible - about 8x slower than the (in my head) equivalent RDD reduceByKey...</p>
  
  <p>My cached DF is just two columns, customer and name with only 50k rows:</p>
  
  <pre><code>== Physical Plan ==
  InMemoryColumnarTableScan [customer#2454,name#2456], InMemoryRelation [customer#2454,name#2456], true, 10000, StorageLevel(true, true, false, true, 1), Scan ParquetRelation[customer#2454,name#2456] InputPaths: hdfs://nameservice1/tmp/v2_selected_parquet/test_parquet2, None
  </code></pre>
  
  <p>When I run the following two snippets, I'd expect similar performance, not the rdd version to run in 10s and the DF version in 85s...</p>
  
  <pre><code>rawtempDF2.rdd.map(lambda x: (x['name'], 1)).reduceByKey(lambda x,y: x+y).collect()
  
  rawtempDF2.groupby('name').count().collect()
  </code></pre>
  
  <p>Am I missing something really fundamental here? FWIW, the RDD version runs 54 stages, and the DF version is 227 :/</p>
  
  <p>Edit: I'm using Spark 1.6.1 and Python 3.4.2. 
  Edit2: Also, the source parquet was partitioned customer/day/name - currently 27 customers, 1 day, c. 45 names.</p>