title: Camel: File consumer component "bites off more than it can chew", pipeline dies from out-of-memory error
body: <p>I have a route defined in Camel that goes something like this: GET request comes in, a file gets created in the file system. File consumer picks it up, fetches data from external web services, and sends the resulting message by POST to other web services.</p>
  
  <p>Simplified code below:</p>
  
  <pre><code>    // Update request goes on queue:
      from("restlet:http://localhost:9191/update?restletMethod=post")
      .routeId("Update via POST")
      [...some magic that defines a directory and file name based on request headers...]
      .to("file://cameldest/queue?allowNullBody=true&amp;fileExist=Ignore")
  
      // Update gets processed
      from("file://cameldest/queue?delay=500&amp;recursive=true&amp;maxDepth=2&amp;sortBy=file:parent;file:modified&amp;preMove=inprogress&amp;delete=true")
      .routeId("Update main route")
      .streamCaching() //otherwise stuff can't be sent to multiple endpoints
      [...enrich message from some web service using http4 component...]
      .multicast()
          .stopOnException()
          .to("direct:sendUpdate", "direct:dependencyCheck", "direct:saveXML")
      .end();
  </code></pre>
  
  <p>The three endpoints in the multicast are simply POSTing the resulting message to other web services.</p>
  
  <p>This all works rather well when the queue (i.e. the file directory <code>cameldest</code>) is fairly empty. Files are being created in <code>cameldest/&lt;subdir&gt;</code>, picked up by the file consumer and moved into <code>cameldest/&lt;subdir&gt;/inprogress</code>, and stuff is being sent to the three outgoing POSTs no problem.</p>
  
  <p>However, once the incoming requests pile up to about 300,000 files progress slows down and eventually the <strong>pipeline fails due to out-of-memory errors</strong> (GC overhead limit exceeded).</p>
  
  <p>By increasing logging I can see that the file consumer polling basically never runs, because it <strong>appears to take responsibility for all files it sees</strong> at each time, waits for them to be done processing, and only then starts another poll round. Besides (I assume) causing the resources bottleneck, this also interferes with my sorting requirements: Once the queue is jammed with thousands of messages waiting to be processed, new messages that would naively be sorted higher up are -if they even still get picked up- still waiting behind those that are already "started".</p>
  
  <p>Now, I've tried the <code>maxMessagesPerPoll</code> and <code>eagerMaxMessagesPerPoll</code> options. They seem to alleviate the problem at first, but after a number of poll rounds I still end up with thousands of files in "started" limbo.</p>
  
  <p>The only thing that sort of worked was making the bottle neck of <code>delay</code> and <code>maxMessages...</code> so narrow that the processing on average would finish faster than the file polling cycle.</p>
  
  <p>Clearly, that is not what I want. I would like my pipeline to process files as fast as possible, but not faster. I was expecting the file consumer to wait when the route is busy.</p>
  
  <p>Am I making an obvious mistake?</p>
  
  <p>(I'm running a somewhat older Camel 2.14.0 on a Redhat 7 machine with XFS, if that is part of the problem.)</p>