title: Does join parallelise well in Spark?
body: <p>I'm running a fairly small Spark program with a few map and reduceByKey operations, over a very small data set of <strong>less than 400MB</strong>.</p>
  
  <p>At some point I have an RDD of tuples that I want to sort, and I call <strong>sortByKey</strong>. This is the slowest part of my program. Everything else seems to run almost instantly, but this takes up to <strong>20 seconds</strong>.</p>
  
  <p>The problem is, it takes 20 seconds in my laptop, as well as in a cluster of AWS m3.large machines. I've tried with 1, 2 and 3 slaves, and the differences in execution time are very small. Ganglia and the spark web console indicate that CPUs and memories are being used to maximum capacity in all slaves, so I think config is ok.</p>
  
  <p>I also found the issue of the execution happening before I expected, but then <a href="http://apache-spark-user-list.1001560.n3.nabble.com/non-lazy-execution-of-sortByKey-td3835.html">I read this thread</a>, which points to an open issue in Spark. I don't think that's entirely related though.</p>
  
  <p>Is it <strong>sortByKey</strong> inherently slow and it doesn't matter how many nodes I add, it's going to dictate the minimum execution time of my program? Hopefully not, and there is just something I'm doing wrong and can be fixed.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Turns out that what I was seeing was related to that link I posted. sortByKey just happened to be the first action (documented as transformation), and it looked as if the program was being slow at sorting, but actually sorting is quite fast. The problem is in a previous <strong>join</strong> operation.</p>
  
  <p>Still everything I said applies by changing sort with join. Why is the execution time not dropping when I add more nodes (or numTask to the join function), and why is it not even better than a plain SQL join? I found <a href="https://groups.google.com/forum/#!topic/spark-users/Ber8KXY2OfU">someone else having this problem</a> before, but no answer other than suggesting tuning serialisation, which I really don't think is my case.</p>