title: Spark on YARN: Less executor memory than set via spark-submit
body: <p>I'm using Spark in a YARN cluster (HDP 2.4) with the following settings:</p>
  
  <ul>
  <li>1 Masternode
  
  <ul>
  <li>64 GB RAM (48 GB usable)</li>
  <li>12 cores (8 cores usable)</li>
  </ul></li>
  <li>5 Slavenodes
  
  <ul>
  <li>64 GB RAM (48 GB usable) each</li>
  <li>12 cores (8 cores usable) each </li>
  </ul></li>
  <li>YARN settings
  
  <ul>
  <li>memory of all containers (of one host): 48 GB</li>
  <li>minimum container size = maximum container size = 6 GB</li>
  <li>vcores in cluster = 40 (5 x 8 cores of workers)</li>
  <li>minimum #vcores/container = maximum #vcores/container = 1</li>
  </ul></li>
  </ul>
  
  <p>When I run my spark application with the command <code>spark-submit --num-executors 10 --executor-cores 1 --executor-memory 5g ...</code> Spark should give each executor 5 GB of RAM right (I set memory only to 5g due to some overhead memory of ~10%).</p>
  
  <p>But when I had a look in the Spark UI, I saw that each executor only has 3.4 GB of memory, see screenshot: </p>
  
  <p><a href="https://i.stack.imgur.com/psvrP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/psvrP.png" alt="screenshot"></a></p>
  
  <p>Can someone explain why there's so less memory allocated?</p>