title: Mongo suffering from a huge number of faults
body: <p>I'm seeing a huge (~200++) faults/sec number in my mongostat output, though very low lock %:</p>
  <p><img src="https://i.stack.imgur.com/MhnFF.png" alt="enter image description here" /></p>
  <p>My Mongo servers are running on m1.large instances on the amazon cloud, so they each have 7.5GB of RAM ::</p>
  <pre><code>root:~# free -tm
               total       used       free     shared    buffers     cached
  Mem:          7700       7654         45          0          0       6848
  </code></pre>
  <p>Clearly, I do not have enough memory for all the cahing mongo wants to do (which, btw, results in huge CPU usage %, due to disk IO).</p>
  <p>I found <a href="http://www.scribd.com/doc/56271132/MongoDB-Performance-Tuning" rel="noreferrer">this document</a> that suggests that in my scenario (high fault, low lock %), I need to &quot;scale out reads&quot; and &quot;more disk IOPS.&quot;</p>
  <p>I'm looking for advice on how to best achieve this. Namely, there are LOTS of different potential queries executed by my node.js application, and I'm not sure where the bottleneck is happening. Of course, I've tried</p>
  <pre><code>db.setProfilingLevel(1);
  </code></pre>
  <p>However, this doesn't help me that much, because the outputted stats just show me slow queries, but I'm having a hard time translating that information into which queries are causing the page faults...</p>
  <p>As you can see, this is resulting in a HUGE (nearly 100%) CPU wait time on my PRIMARY mongo server, though the 2x SECONDARY servers are unaffected...</p>
  <p><img src="https://i.stack.imgur.com/mwBsu.png" alt="enter image description here" /></p>
  <p>Here's what the Mongo docs have to say about page faults:</p>
  <blockquote>
  <p>Page faults represent the number of times that MongoDB requires data not located in physical memory, and must read from virtual memory. To check for page faults, see the extra_info.page_faults value in the serverStatus command. This data is only available on Linux systems.</p>
  <p>Alone, page faults are minor and complete quickly; however, in aggregate, large numbers of page fault typically indicate that MongoDB is reading too much data from disk and can indicate a number of underlying causes and recommendations. In many situations, MongoDB’s read locks will “yield” after a page fault to allow other processes to read and avoid blocking while waiting for the next page to read into memory. This approach improves concurrency, and in high volume systems this also improves overall throughput.</p>
  <p>If possible, increasing the amount of RAM accessible to MongoDB may help reduce the number of page faults. If this is not possible, you may want to consider deploying a shard cluster and/or adding one or more shards to your deployment to distribute load among mongod instances.</p>
  </blockquote>
  <p>So, I tried the recommended command, which is terribly unhelpful:</p>
  <pre><code>PRIMARY&gt; db.serverStatus().extra_info
  {
      &quot;note&quot; : &quot;fields vary by platform&quot;,
      &quot;heap_usage_bytes&quot; : 36265008,
      &quot;page_faults&quot; : 4536924
  }
  </code></pre>
  <p>Of course, I could increase the server size (more RAM), but that is expensive and seems to be overkill. I should implement sharding, but I'm actually unsure what collections need sharding! Thus, I need a way to isolate where the faults are happening (what specific commands are causing faults).</p>
  <p>Thanks for the help.</p>