title: Spark best practices for reduceByKey
body: <p>I have a dataframe with next schema:</p>
  
  <pre><code>root
   |-- id_1: long (nullable = true)
   |-- id_2: long (nullable = true)
   |-- score: double (nullable = true)
  </code></pre>
  
  <p>The data looks like :</p>
  
  <pre><code>+----+----+------------------+
  |id_1|id_2|score             |
  +----+----+------------------+
  |0   |9   |0.5888888888888889|
  |0   |1   |0.6166666666666667|
  |0   |2   |0.496996996996997 |
  |1   |9   |0.6222222222222221|
  |1   |6   |0.9082996632996633|
  |1   |5   |0.5927450980392157|
  |2   |3   |0.665774107440774 |
  |3   |8   |0.6872367465504721|
  |3   |8   |0.6872367465504721|
  |5   |6   |0.5365909090909091|
  +----+----+------------------+
  </code></pre>
  
  <p>The goal is to <strong>find id_2 for each id_1 with max score</strong>. Maybe I'm wrong, but... just need to create paired RDD:</p>
  
  <pre><code>root
   |-- _1: long (nullable = true)
   |-- _2: struct (nullable = true)
   |    |-- _1: long (nullable = true)
   |    |-- _2: double (nullable = true)
  
  +---+----------------------+
  |_1 |_2                    |
  +---+----------------------+
  |0  |[9,0.5888888888888889]|
  |0  |[1,0.6166666666666667]|
  |0  |[2,0.496996996996997] |
  |1  |[9,0.6222222222222221]|
  |1  |[6,0.9082996632996633]|
  |1  |[5,0.5927450980392157]|
  |2  |[3,0.665774107440774] |
  |3  |[8,0.6872367465504721]|
  |3  |[8,0.6872367465504721]|
  |5  |[6,0.5365909090909091]|
  +---+----------------------+
  </code></pre>
  
  <p>and reduce by key with max. Something like</p>
  
  <pre><code>paired_rdd.reduceByKey(lambda x1, x2: max(x1, x2, key=lambda x: x[-1]))
  </code></pre>
  
  <p>Or the same with DataFrame API (without paired rdd):</p>
  
  <pre><code>original_df.groupBy('id_1').max('score')
  </code></pre>
  
  <p>I have two questions and will be appreciated if someone can point on my wrong steps.</p>
  
  <ol>
  <li><p>For 1 billion or even 100 billion records: what are best practices to achieve the goal (<em>find id_2 for each id_1 with max score</em>)? I've tried with 50 million and 100M records and had better results with shuffling the data (which is opposite to what Holden Karau told). I've done repartitioning by <code>id_1</code></p>
  
  <p>.repartition(X, "id_1")</p>
  
  <p>then <strong>reduceByKey</strong> and it was faster. Why?</p></li>
  <li><p>Why DataFrame API was slower several times vs RDD API? Where I'm wrong?</p></li>
  </ol>
  
  <p>Thanks.</p>