title: spark on yarn, Diagnostics: Container killed on request. Exit code is 143
body: <p>it's a simple spark writen by python. I got "Diagnostics: Container killed on request. Exit code is 143" error. Here is my code and script to run this code.<br> 
  The size of input dir is roughly 100GB. But if I use a small data file(3GB), it will work well.</p>
  
  <pre><code>import sys
  from pyspark import SparkContext
  
  sc = SparkContext(appName="job_name")
  
  data1 = sc.textFile(sys.argv[1])
  d1 = data1.filter(lambda x: "a string" in x)
  print d1.count()
  
  sc.stop()
  
  //-----------------------------------------//
  
  input="xxxxx"
  output="yyyyy"
  
  hadoop fs -rmr $output
  $SPARK_HOME/bin/spark-submit \
    --deploy-mode cluster \
    --master yarn \
    --num-executors 100 \
    --executor-cores 2 \
    --driver-cores 2 \
    --executor-memory 8g \
    --driver-memory 4g \
    stat.py \
    $input \
    $output
  </code></pre>
  
  <p>So I wrote a hadoop streaming to do this job, and I got:</p>
  
  <pre><code>Error: Java heap space
  Container killed by the ApplicationMaster.
  Container killed on request. Exit code is 143
  Container exited with a non-zero exit code 143
  </code></pre>
  
  <p>it seems to be the same error with my spark code. </p>
  
  <p>Now I suspect that there are maybe some records in input is too large. </p>
  
  <p>/* <br>
  * add at 2016-12-16<br>
  */ <br>
  my hadoop streaming is working ^-^, with config like this:</p>
  
  <pre><code>hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.5.0-cdh5.3.2.jar
     -D mapred.reduce.tasks=0 \
     -D mapreduce.map.memory.mb=3192 \
     -D mapreduce.reduce.memory.mb=3192 \
     -D mapreduce.map.java.opts=-Xmx2872m \
     -D mapreduce.reduce.java.opts=-Xmx2872m \
     -D mapred.child.java.opts=-Xmx2048m \
     -D mapreduce.task.io.sort.mb=512 \
     -mapper "mapper.py" \
     -file mapper.py \
     -input $input \
     -output $output
  </code></pre>
  
  <p>but my spark code still can not run, no matter how i tune it.
  this is my config of spark code:</p>
  
  <pre><code>$SPARK_HOME/bin/spark-submit \
      --deploy-mode cluster \
      --master yarn \
      --num-executors 200 \
      --executor-cores 2 \
      --driver-cores 2 \
      --executor-memory 20g \
      --driver-memory 16g \
      --verbose \
      stat.py \
      $input \
      $output
  
  Here is my spark-defaults.conf:
  spark.yarn.executor.memoryOverhead      4096
  spark.yarn.driver.memoryOverhead        4096
  spark.memory.storageFraction    0.9
  </code></pre>
  
  <p>With these infomation, does anyone know how to tune spark.</p>