title: MongoDB is giving inconsistent write times
body: <p>I am using Scala, Reactive Mongo 0.10.5 and Mongo 2.6.4 running on Ubuntu.  I have tested on a few machine configurations but right now I am working with 15gb of memory, 2 cores and 60gb of SSD storage (AWS)</p>
  <p>I have just set up a test mongo instance and have been using it to benchmark a few things, however I am seeing some inconsistency that I can't explain.</p>
  <p>I am writing a consistent amount of data using 10 separate threads to a single collection.  Each write consists of a document containing an array which contains 1000 elements.  Each element is a complex document consisting of several fields and nested fields.  I have tested with arrays of 1000, 10000 and 100 and have seen the same behavior with all.  Each write is unique (i.e. I never write to the same document twice)</p>
  <p>The write speed tends to be around 100-200ms per write with the current hardware I am using.  I would like better but that isn't my main issue.</p>
  <p>My main issue is that sometimes the write times will spike.  When they do, it can take a single write several seconds to complete.  They do eventually complete but it takes a while.  I have timeouts built into the app doing the writing (10 seconds) and when the spikes happen it will frequently hit that timeout.  I have increased the timeout and verified that the write does eventually complete but it can take a long time (30+ seconds).</p>
  <p>I have worked with Mongo before using the Mongo Java Driver in Scala and have not noticed this problem.  However it is unclear whether the issue is a result of the driver, or my Mongo setup.</p>
  <p>I have looked at the logs and while they report when the query is taking longer, they don't actually provide any information about why it is taking longer.  I have done the same with profiling and again they report a long query but don't say why it is long.</p>
  <p>I have run mongostat while running and it seems that when the writes start taking a long time I notice a similar slow down in mongostat.  I.E. mongostat will pause for several seconds before continuing.</p>
  <p>The mongo machine itself is bored while this is happening.  Load averages are minimal as are CPU and memory usage.  It does not appear to be going into swap.</p>
  <p>I suspect I just have something configured incorrectly in the Mongo but I haven't been able to find anything that indicates what.</p>
  <p>Has anyone seen this behavior before?  Is it something in my configuration or perhaps something with the Reactive Mongo driver?</p>
  <h1>UPDATE:</h1>
  <p>Using iostat I was able to determine that the normal writes/second is hitting around 1Mb/second.  However during the slow periods it spikes to 6-7Mb/second.</p>
  <p>I also found the following in the mongo logs.</p>
  <blockquote>
  <p>[DataFileSync] flushing mmaps took 15621ms  for 35 files</p>
  <p>[DataFileSync] flushing mmaps took 14816ms  for 22 files</p>
  </blockquote>
  <p>In at least one case this log statement corresponds exactly with one of the slow downs.</p>
  <p>This definitely seems to be a disk flush problem based on these observations.</p>
  <p>Does this imply that I am pushing more data than the current Mongo configuration can handle?  Or is there some other configuration that can be done to reduce the impact of those flushes?</p>