title: Slave lost error in pyspark
body: <p><a href="https://i.stack.imgur.com/nnIf9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nnIf9.png" alt="picture from spark UI"></a></p>
  
  <p>I'm using Spark1.6
  I'm running a simple df.show(2) method and got errors like  </p>
  
  <pre><code>    An error occurred while calling o143.showString.
      : org.apache.spark.SparkException: Job aborted due to stage      
      failure: Task 6 in stage 6.0 failed 4 times, most recent failure:
       Lost task 6.3 in stage 6.0 
       ExecutorLostFailure (executor 2 exited caused by one of the 
       running tasks) Reason: Slave lost
  </code></pre>
  
  <p>When I did persist, through spark UI I saw the shuffleWrite memory is very high and took a long time and still returned errors.
  Through some search, I found these might be the out of memory problem.
  Following this link <a href="https://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space/22742982#22742982">out of memory error Java</a>
  I  did a repartition up to 1000, still not so helpful.</p>
  
  <p>I set up the SparkConf as</p>
  
  <pre><code>     conf = (SparkConf().set("spark.driver.maxResultSize", "150g").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")) 
  </code></pre>
  
  <p>My server side memory could be up to 200GB </p>
  
  <p>Do yo have any good idea to do this or point me to related links. Pyspark will be most helpful</p>
  
  <p>Here is the error log from YARN:</p>
  
  <pre><code>     Application application_1477088172315_0118 failed 2 times due to
       AM Container for appattempt_1477088172315_0118_000006 exited 
       with exitCode: 10
  
       For more detailed output, check application tracking page: Then, 
       click on links to logs of each attempt.
       Diagnostics: Exception from container-launch.
       Container id: container_1477088172315_0118_06_000001
       Exit code: 10
       Stack trace: ExitCodeException exitCode=10:
       at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
       at org.apache.hadoop.util.Shell.run(Shell.java:479)
       at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
       at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
  
  
  Container exited with a non-zero exit code 10
  Failing this attempt. Failing the application.
  </code></pre>
  
  <p>Here is the error info from notebook:</p>
  
  <p>Py4JJavaError: An error occurred while calling o71.showString.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 15.0 failed 4 times, most recent failure: Lost task 1.3 in stage 15.0 (): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Slave lost
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
      at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:212)
      at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:165)
      at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:174)
      at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499)
      at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
      at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2086)
      at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$execute$1(DataFrame.scala:1498)
      at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$collect(DataFrame.scala:1505)
      at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1375)
      at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1374)
      at org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:2099)
      at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1374)
      at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1456)
      at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:170)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
      at py4j.Gateway.invoke(Gateway.java:259)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:209)
      at java.lang.Thread.run(Thread.java:745)</p>
  
  <p>Thank you</p>