title: Python Multiprocessing Reading from SQS
body: <p>I have the following code that launches multiple python processes that continually poll from an SQS queue.</p>
  
  <p>The processes are launched with </p>
  
  <pre><code>num_processes = range(1, 9)
  
      for p_num in num_processes:
          p = multiprocessing.Process(
              target=sqs_polling, args=(queue_name, p_num,))
          p.start()
  </code></pre>
  
  <p>and the actual polling function is</p>
  
  <pre><code>def sqs_polling(queue_name, process_id):
  
      sqs = boto3.resource('sqs', region_name='us-east-1')
      queue = sqs.get_queue_by_name(QueueName=queue_name)
  
      no_messages = False
  
      # poll sqs forever
      while 1:
  
          # polling delay so aws does not throttle us
          sleep(2.0)
  
          # sleep longer if there are no messages on the queue the last time it was polled
          if no_messages:
              sleep(900.0)
  
          message_batch = queue.receive_messages(MaxNumberOfMessages=10, WaitTimeSeconds=20)
  
          if len(message_batch) == 0:
              no_messages = True
          else:
              no_messages = False
  
          # process messages
          for message in message_batch:
              do_something(message)
              message.delete()
  </code></pre>
  
  <p>This seems to work for a few hours but eventually it seems as though SQS throttles the processes and no messages can be read even though they exist on the queue.  To help reduce this I have a timeout of 2 seconds between Queue reads.  Also I have created a 15min timeout if there are no messages read.  In spite of this I still get throttling.  Can anyone explain why throttling is still occurring here?  Another possibility might be that the connection to the queue gets stale but I think that is unlikely.</p>