title: Spark: difference of semantics between reduce and reduceByKey
body: <p>In Spark's documentation, it says that RDDs method <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD" rel="nofollow"><code>reduce</code></a> requires a associative AND commutative binary function.</p>
  
  <p>However, the method <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions" rel="nofollow"><code>reduceByKey</code></a> ONLY requires an associative binary function.</p>
  
  <pre><code>sc.textFile("file4kB", 4)
  </code></pre>
  
  <p>I did some tests, and apparently it's the behavior I get. Why this difference? Why does <code>reduceByKey</code> ensure the binary function is always applied in certain order (to accommodate for the lack of commutativity) when <code>reduce</code> does not?</p>
  
  <p>Example, if a load some (small) text with 4 partitions (minimum):</p>
  
  <pre><code>val r = sc.textFile("file4k", 4)
  </code></pre>
  
  <p>then:</p>
  
  <pre><code>r.reduce(_ + _)
  </code></pre>
  
  <p>returns a string where parts are not always in the same order, whereas:</p>
  
  <pre><code>r.map(x =&gt; (1,x)).reduceByKey(_ + _).first
  </code></pre>
  
  <p>always returns the same string (where everything is in the same order than in the original file).</p>
  
  <p>(I checked with <code>r.glom</code> and the file content is indeed spread over 4 partitions, there is no empty partition).</p>