Answers: <p>Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-06-09T07:10:23.883Z
Answers: 2020-06-09T07:10:23.883Z
Answers: anVzdGFub3RoZXJodW1hbg
Answers: <p><strong>Disclaimer</strong>: this is <em>certainly not the best solution</em> for backend workers, I'm just trying to be creative within the given constraints.</p>
  
  <p>First, you'll need to be familiar with SQS's visibility timeout (<a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html" rel="nofollow noreferrer">documentation</a>) for this to work.</p>
  
  <p>Let's assume the following, then I'll walk through how it might work:</p>
  
  <ul>
  <li>We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)</li>
  <li>We can take up to 12 minutes to process a message in the worst-case scenario</li>
  </ul>
  
  <p>Next, some configuration changes: </p>
  
  <ul>
  <li>We configure the ASG to scale desired count to 0 when <code>ApproximateNumberOfMessagesVisible</code> = 0 for 5 minutes.</li>
  <li>We configure the ASG to scale desired count to 1 when <code>ApproximateNumberOfMessagesVisible</code> >= 1.</li>
  <li>We set all messages <code>VisibilityTimeout</code> to 10 minutes.</li>
  </ul>
  
  <p>A failure scenario might work out like this:</p>
  
  <ol>
  <li>At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, <code>ApproximateNumberOfMessagesVisible</code> = 0</li>
  <li>At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.</li>
  <li>At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)</li>
  <li>At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. <code>ApproximateNumberOfMessagesVisible</code> = 1</li>
  <li>Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)</li>
  </ol>
  
  <p>The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.</p>
  
  <p>Reference:</p>
  
  <p><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html</a></p>
  
  <p><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html</a></p>
  
  <p><strong>HOWEVER...</strong></p>
  
  <p>The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-06-10T22:41:30.180Z
Answers: Thank you! I need to think about this a little bit. The main problem is throughput. The 10 messages per day usually come in in 1 batch or 2 batches, so this will cause a lot of throughput delay.
Answers: jdog
Answers: CC BY-SA 4.0
Answers: 2020-06-12T17:44:39.190Z
Answers: Hmm, I suppose my hacky solution would work if instead of scaling to 1 instance you scaled to N, where N is the most batches you would see at one time. It's likely to be wasteful (in the sense that it over-provisions at times) but at least it's still cheaper than running 1 full-time like you are now.
Answers: Woodland
Answers: CC BY-SA 4.0
Answers: 2020-06-15T21:07:56.947Z
Answers: The answer clarified/ reminded me that I can simply terminate my instance when messages have been in the queue for too long, because I can expect all messages to be processed within a certain timeframe based on my assumption of large gaps in processing traffic. Even with consistent processing traffic, my processing server would simply be terminated and rebuild every x hours.
Answers: jdog
Answers: CC BY-SA 4.0
Answers: 2020-06-15T21:09:42.633Z
Answers: there is another piece of technical debt in this system, which is that the processing is simply running a macro in Excel triggered by Windows task scheduler. Because of its minimum 5 minute time resolution, processing doesn't start on average for 2.5 minutes after boot up is complete. Windows instances being billed at 1 hour minimum also complicate the trade off.
Answers: jdog
Answers: CC BY-SA 4.0
Answers: 2020-06-16T17:47:46.317Z
Answers: Good point, especially the Windows billing consideration. Sounds like quite the difficult system, I don't envy you!
Answers: Woodland
Answers: CC BY-SA 4.0
Answers: 2020-06-16T21:14:45.747Z
Answers: not too bad. Its essentially a calculator used for marketing, not really mission critical. It could be done better for more budget, but there are always constraints
Answers: jdog
Answers: CC BY-SA 4.0
Answers: 2020-06-10T19:03:40.357Z
Answers: 2020-06-10T19:03:40.357Z
Answers: Woodland
Body: <p>I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process.</p>
  
  <p>Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but keep the autoscaling size to 1, as the problem is often resolved with a fresh server created by autoscaling.</p>
  
  <p>Can this be done with autoscaling configuration alone, or do I need to create a separate process with step or lambda functions?</p>
  
Tags: <autoscaling><amazon-sqs>
Title: Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1
UserDisplayName: jdog
ThreadText: # Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1
  
  (Asked by: username_0 on 2020-06-04)
  
  I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process.
  
  Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but keep the autoscaling size to 1, as the problem is often resolved with a fresh server created by autoscaling.
  
  Can this be done with autoscaling configuration alone, or do I need to create a separate process with step or lambda functions?
  
  **Tags:** autoscaling, amazon-sqs
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-06-10. Score: 4 | Accepted answer
  
  **Disclaimer**: this is *certainly not the best solution* for backend workers, I'm just trying to be creative within the given constraints.
  
  First, you'll need to be familiar with SQS's visibility timeout ([documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html)) for this to work.
  
  Let's assume the following, then I'll walk through how it might work:
  
  * We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
  * We can take up to 12 minutes to process a message in the worst-case scenario
  
  Next, some configuration changes:
  
  * We configure the ASG to scale desired count to 0 when `ApproximateNumberOfMessagesVisible` = 0 for 5 minutes.
  * We configure the ASG to scale desired count to 1 when `ApproximateNumberOfMessagesVisible` >= 1.
  * We set all messages `VisibilityTimeout` to 10 minutes.
  
  A failure scenario might work out like this:
  
  1. At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, `ApproximateNumberOfMessagesVisible` = 0
  2. At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
  3. At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
  4. At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. `ApproximateNumberOfMessagesVisible` = 1
  5. Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)
  
  The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.
  
  Reference:
  
  <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html>
  
  <https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html>
  
  **HOWEVER...**
  
  The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!
  
  > **Comments:**
  > **username_0**: Thank you! I need to think about this a little bit. The main problem is throughput. The 10 messages per day usually come in in 1 batch or 2 batches, so this will cause a lot of throughput delay.  
  > **username_2**: Hmm, I suppose my hacky solution would work if instead of scaling to 1 instance you scaled to N, where N is the most batches you would see at one time. It's likely to be wasteful (in the sense that it over-provisions at times) but at least it's still cheaper than running 1 full-time like you are now.  
  > **username_0**: The answer clarified/ reminded me that I can simply terminate my instance when messages have been in the queue for too long, because I can expect all messages to be processed within a certain timeframe based on my assumption of large gaps in processing traffic. Even with consistent processing traffic, my processing server would simply be terminated and rebuild every x hours.  
  > **username_0**: there is another piece of technical debt in this system, which is that the processing is simply running a macro in Excel triggered by Windows task scheduler. Because of its minimum 5 minute time resolution, processing doesn't start on average for 2.5 minutes after boot up is complete. Windows instances being billed at 1 hour minimum also complicate the trade off.  
  > **username_2**: Good point, especially the Windows billing consideration. Sounds like quite the difficult system, I don't envy you!  
  > **username_0**: not too bad. Its essentially a calculator used for marketing, not really mission critical. It could be done better for more budget, but there are always constraints  
  
  ---
  
  ### Answer by username_1 on 2020-06-09. Score: 1
  
  Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.
  
  
  ---
  
  