title: RabbitMQ consumer utilization zero under high load
body: <p>We use spring amqp based listener container - with about 40 consumers, prefetch count 1. The message ttl is about 60sec before it gets to dead letter queue.</p>
  
  <p>The operation performed by each consumer is a database update - which is slower than the rate at which the messages arrive into the queue.</p>
  
  <p>After sometime, the number of messages pile up in queue the consumer utilization is down to zero. I was under the impression that the consumers are blocked on database. However if I look at the thread dump all the consumers are in wait state on rabbit mq - no messages are being processed.</p>
  
  <pre><code>    "SimpleAsyncTaskExecutor-7" #51 prio=5 os_prio=0 tid=0x00007fcb01ad0800 nid=0x58f7 waiting on condition [0x00007fcae5af1000]
     java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  &lt;0x00000000854c30c8&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:390)
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1097)
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1086)
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1100(SimpleMessageListenerContainer.java:93)
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1203)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>The queue also gets into flow state. </p>
  
  <p>Not sure why the queue message processing is stopped. I understand publish to it being restricted.</p>
  
  <p>Any suggestions would help.</p>