title: java.lang.OutOfMemoryError: Java heap space using Docker
body: <p>So I am running the following locally (standalone):</p>
  
  <pre><code>~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --py-files afile.py run_script.py 
  </code></pre>
  
  <p>And I got the following error:</p>
  
  <pre><code>java.lang.OutOfMemoryError: Java heap space
  </code></pre>
  
  <p>To overpass this I am running the following:</p>
  
  <pre><code>~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --driver-memory 6G --executor-memory 1G --py-files afile.py run_script.py
  </code></pre>
  
  <p>and the script runs normally.</p>
  
  <p>Now, I am using the following <a href="https://github.com/gettyimages/docker-spark" rel="nofollow noreferrer">docker build for Spark</a> and run the following:</p>
  
  <pre><code>docker-compose up
  docker exec app_master_1 bin/spark-submit --driver-memory 6G --executor-memory 1G --py-files afile.py run_script.py
  </code></pre>
  
  <p>In that case I still get the error of:</p>
  
  <pre><code>2018-06-13 21:43:16 WARN  TaskSetManager:66 - Lost task 0.0 in stage 3.0 (TID 9, 172.17.0.3, executor 0): java.lang.OutOfMemoryError: Java heap space
  at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:77)
  at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:219)
  at org.apache.spark.sql.execution.datasources.text.TextFileFormat$$anonfun$readToUnsafeMem$1$$anonfun$apply$4.apply(TextFileFormat.scala:143)
  at org.apache.spark.sql.execution.datasources.text.TextFileFormat$$anonfun$readToUnsafeMem$1$$anonfun$apply$4.apply(TextFileFormat.scala:140)
  at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
  at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
  at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.next(FileScanRDD.scala:109)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
  at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
  at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.fold(TraversableOnce.scala:212)
  at scala.collection.AbstractIterator.fold(Iterator.scala:1336)
  at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$19.apply(RDD.scala:1090)
  at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$19.apply(RDD.scala:1090)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2123)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2123)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:109)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>and somewhere later:</p>
  
  <pre><code>2018-06-13 21:43:17 ERROR TaskSchedulerImpl:70 - Lost executor 0 on 172.17.0.3: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages
  </code></pre>
  
  <p>As far as I undersand even though its written that its out of memory in executor 0 i have to increase the driver-memory as its a standalone, right?? </p>
  
  <p>Any idea why is this happening and how to overpass it?</p>
  
  <p><em>edit</em></p>
  
  <p>The error is happening when i am trying to use <code>sqlCont.read.json(json_path)</code> where the file is not even big enough.</p>