title: How to set Apache Spark Executor memory
body: <p>How can I increase the memory available for Apache spark executor nodes?</p>
  
  <p>I have a 2 GB file that is suitable to loading in to Apache Spark. I am running apache spark for the moment on 1 machine, so the driver and executor are on the same machine. The machine has 8 GB of memory. </p>
  
  <p>When I try count the lines of the file after setting the file to be cached in memory I get these errors:</p>
  
  <pre><code>2014-10-25 22:25:12 WARN  CacheManager:71 - Not enough space to cache partition rdd_1_1 in memory! Free memory is 278099801 bytes.
  </code></pre>
  
  <p>I looked at the documentation <a href="http://spark.apache.org/docs/latest/configuration.html" rel="noreferrer">here</a> and set <code>spark.executor.memory</code> to <code>4g</code> in <code>$SPARK_HOME/conf/spark-defaults.conf</code></p>
  
  <p>The UI shows this variable is set in the Spark Environment. You can find screenshot <a href="https://drive.google.com/file/d/0B0B_O5bxDDlsc3JmY0xfbjFtN0k/view?usp=sharing" rel="noreferrer">here</a></p>
  
  <p>However when I go to the <a href="https://drive.google.com/file/d/0B0B_O5bxDDlsV25ka2lHd1ZFNzA/view?usp=sharing" rel="noreferrer">Executor tab</a> the memory limit for my single Executor is  still set to 265.4 MB.  I also still get the same error. </p>
  
  <p>I tried various things mentioned <a href="https://stackoverflow.com/questions/24242060/how-to-change-memory-per-node-for-apache-spark-worker">here</a> but I still get the error and don't have a clear idea where I should change the setting.  </p>
  
  <p>I am running my code interactively from the spark-shell</p>