title: How does Hadoop allocate memory?
body: <p>Please help me understand ... I am using a very memory-intensive algorithm (LDA on Mahout). Because it requires huge amounts of memory, I used a 5 node Hadoop cluser with 24GB of RAM each.</p>
  
  <p>However, when running the algorithm, I get this error:
  Container [pid=14275,containerID=container_1458474415429_0004_01_000012] is running beyond virtual memory limits. Current usage: 265.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.</p>
  
  <p>Help me understand this ... How does Hadoop allocate memory? How should I set up the environment to use the maximum possible memory available? I believed Hadoop will just break up the tasks into smaller ones, therefore each of them using less memory - however, that seems naive now.</p>