I have deployed a python-flask web app on the worker tier of AWS. I send some data into the associated SQS queue and the daemon forwards the request data in a POST request to my web app. The web app takes anywhere between 5 mins to 6 hours to process the request depending upon the size of posted data. I have also configured the worker app into an auto scaling group to scale based on CPU utilization metrics. When I send 2 messages to the queue in quick succession, both messages start showing up as in-flight. I was hoping that the daemon will forward the first message to the web app and then wait for it to be processed before pulling the second message out. In the meantime, auto scaling will spin up another instance (which it is but since the second message is also in-flight, it is not able to pull that message) and the new instance will pull and process the second message. Is there a way of achieving this?
Set the HTTP Connection setting under Worker Configuration to 1. This should prevent each server from receiving more than 1 message at a time.
You might want to look into changing your autoscaling configuration to monitor your SQS queue depth or some other SQS metric instead of worker CPU utilization.
AWS worker daemon locks multiple messages even before the first message is processed