title: Container is running beyond memory limits
body: <p>In Hadoop v1, I have assigned each 7 mapper and reducer slot with size of 1GB, my mappers &amp; reducers runs fine. My machine has 8G memory, 8 processor. 
  Now with YARN, when run the same application on the same machine, I got container error. 
  By default, I have this settings:</p>
  
  <pre><code>  &lt;property&gt;
      &lt;name&gt;yarn.scheduler.minimum-allocation-mb&lt;/name&gt;
      &lt;value&gt;1024&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;yarn.scheduler.maximum-allocation-mb&lt;/name&gt;
      &lt;value&gt;8192&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
      &lt;value&gt;8192&lt;/value&gt;
    &lt;/property&gt;
  </code></pre>
  
  <p>It gave me error:</p>
  
  <pre><code>Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
  </code></pre>
  
  <p>I then tried to set memory limit in mapred-site.xml:</p>
  
  <pre><code>  &lt;property&gt;
      &lt;name&gt;mapreduce.map.memory.mb&lt;/name&gt;
      &lt;value&gt;4096&lt;/value&gt;
    &lt;/property&gt;
    &lt;property&gt;
      &lt;name&gt;mapreduce.reduce.memory.mb&lt;/name&gt;
      &lt;value&gt;4096&lt;/value&gt;
    &lt;/property&gt;
  </code></pre>
  
  <p>But still getting error:</p>
  
  <pre><code>Container [pid=26783,containerID=container_1389136889967_0009_01_000002] is running beyond physical memory limits. Current usage: 4.2 GB of 4 GB physical memory used; 5.2 GB of 8.4 GB virtual memory used. Killing container.
  </code></pre>
  
  <p>I'm confused why the the map task need this much memory. In my understanding, 1GB of memory is enough for my map/reduce task. Why as I assign more memory to container, the task use more? Is it because each task gets more splits? I feel it's more efficient to decrease the size of container a little bit and create more containers, so that more tasks are running in parallel. The problem is how can I make sure each container won't be assigned more splits than it can handle?   </p>