title: Which type of Spark memory should I increase on Java out of memory error?
body: <p>So, I have a pattern like shown below.</p>
  
  <pre><code>def someFunction(...) : ... = 
  {
    // Somewhere here some large string (still &lt; 1 GB) is made ...
    //  ... and sometimes I get Java.lang.OutOfMemoryError while building that string
  }
  
  ....
  val RDDb = RDDa.map(x =&gt; someFunction(...))
  </code></pre>
  
  <p>So, inside <code>someFunction</code>, at one place a large string is made, which is still not that big (&lt; 1 GB), but I get <code>java.lang.OutOfMemoryError: Java heap space</code> error sometimes while building that string. This happens even when my executor memory is quite large (8 GB). </p>
  
  <p>According to <a href="https://0x0fff.com/spark-memory-management/" rel="nofollow noreferrer">this article</a>, there is User memory and Spark memory. Now in my case, which one's fraction should I increase, the User memory's or the Spark memory's? </p>
  
  <p><strong>P.S: I am using Spark version 2.0</strong></p>