title: Extremely Slow Processing on Dataproc 9 hours vs 3 mins on local machine
body: <p>From the log I can see that there are 182k rows 70MB. It take 1.5 hours  load 70MB of data and 9 hours (Started at 15/11/14 01:58:28  and ended at 15/11/14 09:19:09) to train 182K rows on Dataproc. Loading the same data and running the same algorithm on my local machine takes 3 mins</p>
  
  <p>DataProc Log</p>
  
  <pre><code>15/11/13 23:27:09 INFO com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage: Table 'mydata-data:website_wtw_feed.video_click20151111' to be exported has 182712 rows and 70281790 bytes
  15/11/13 23:28:13 WARN akka.remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@rc-spark-poc-w-1.c.dailymotion-data.internal:60749] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
  
  15/11/14 01:58:28 INFO com.dailymotion.recommender.BigQueryRecommender: Fetching the Ratings RDD
  15/11/14 01:58:28 INFO com.dailymotion.recommender.BigQueryRecommender: Transforming the video feature matrix
  15/11/14 01:58:28 INFO com.dailymotion.recommender.BigQueryRecommender: Training ALS Matrix factorization Model
  
  
  [Stage 2:=============================&gt;                             (1 + 1) / 2]
  
  15/11/14 09:19:09 WARN com.github.fommil.netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
  15/11/14 09:19:09 WARN com.github.fommil.netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
  
  15/11/14 09:19:44 INFO com.dailymotion.recommender.BigQueryRecommender: Transforming the video feature matrix
  15/11/14 09:19:44 INFO com.dailymotion.recommender.BigQueryRecommender: Transforming the user feature matrix
  </code></pre>
  
  <ol>
  <li><p>Copied the data to local machine</p>
  
  <pre><code>r.viswanadha$ gsutil cp -r gs://&lt;mycompany&gt;-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000 .
  
  
  Copying gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/data-000000000000.json... 
  
  Downloading ...201511132327_0000/shard-0/data-000000000000.json: 141.3 MiB/141.3 MiB      
  
  Copying gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/data-000000000001.json... 
  
  Copying gs://&lt;mycompany&gt;-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-1/data-000000000000.json...`
  </code></pre></li>
  <li><p>Ran the same Algorithm. ALS Train step took ~3 mins</p>
  
  <pre><code>com.dailymotion.recommender.BigQueryRecommender --app_name BigQueryRecommenderTest --master local[4] --input_dir /Users/r.viswanadha/Documents/workspace/rec-spark-java-poc/input/job_201511132327_0000/shard-0/ 
  </code></pre></li>
  </ol>
  
  <p>First Run</p>
  
  <pre><code>15/11/14 13:19:36 INFO BigQueryRecommender: Training implicit features for the ALS Matrix factorization Model
  ...
  15/11/14 13:22:24 INFO BigQueryRecommender: Transforming the video feature matrix
  </code></pre>
  
  <p>Second Run</p>
  
  <pre><code>15/11/14 13:29:05 INFO BigQueryRecommender: Training implicit features for the ALS Matrix factorization Model
  
  
  ...
  
  15/11/14 13:31:57 INFO BigQueryRecommender: Transforming the video feature matrix
  </code></pre>
  
  <p>The DataProc cluster has  1 Master and 3 Slaves with 104GB (RAM) and 16 CPUs each.</p>
  
  <p>My local machine has 8GB (RAM) and 2 CPU 2.7GHz Core i5</p>
  
  <pre><code>gsutil ls -l -r -h  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000
  
  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/: 
  
  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/: 
  
      0 B  2015-11-13T23:27:13Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/ 
  
      141.3 MiB  2015-11-13T23:29:21Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/data-000000000000.json 
  
     0 B  2015-11-13T23:29:21Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-0/data-000000000001.json 
  
  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-1/: 
  
      0 B  2015-11-13T23:27:13Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-1/ 
  
      0 B  2015-11-13T23:28:47Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/shard-1/data-000000000000.json 
  
     0 B  2015-11-13T23:27:09Z  gs://dailymotion-spark-rc-test/bqdata/hadoop/tmp/bigquery/job_201511132327_0000/ 
  
  TOTAL: 6 objects, 148165416 bytes (141.3 MiB)
  </code></pre>