title: Low performance reduceByKey() in Spark
body: <p>I am writing a program on Spark, which I just do the aggregate by the key. The program is pretty simple. My input data is only 2GB, running on a multi-core server (8cores, 32GB RAM) with the setting local[2]. That is using two cores for parallelization. However, I found that the performance is pretty bad. It almost takes two hours to complete. I am using KryoSerializer. I guess it might be caused by Serializer. How to solve this problem?</p>
  
  <pre><code>  val dataPoints = SparkContextManager.textFile(dataLocation)
          .map(x =&gt; {
              val delimited = x.split(",")
              (delimited(ColumnIndices.HOME_ID_COLUMN).toLong, 
                  delimited(ColumnIndices.USAGE_READING_COLUMN).toDouble)
          })
  
  def process(step: Int): Array[(Long, List[Double])] = {
    val resultRDD = new PairRDDFunctions(dataPoints.map(x =&gt;(x._1, List[Double](x._2))))
    resultRDD.reduceByKey((x, y) =&gt; x++y).collect()
  }
  </code></pre>
  
  <p>The output will be:</p>
  
  <pre><code>1, [1, 3, 13, 21, ..., 111] // The size of list is about 4000
  2, [24,34,65, 24, ..., 245]
  ....
  </code></pre>