Body: <p>We have a range of AWS/NatGateway alerts set up to monitor the health of our networking.  When the cloudwatch alerts fire, this goes to an SNS/Lambda, which then fires a message to Slack. One of the alerts is an ActiveConnectionCount, which fires if the nat gateway has zero connections.  However, outside of regular office hours, it is really common for zero connections.  What this leads to is dozens of messages sitting in Slack on Monday morning, and there is no problem to resolve.</p>
  <p>While this specific alert was an issue, it has led to a wider question: Has anyone found a good solution to manage cloudwatch alerting out of hours? I haven't found a way to set a time in cloudwatch to run the alerts, such as Mon-Fri 8am-6pm, i don't think it's possible.  I could set a cron job to disable and enable alert actions which could just disable alerts outside office hours, but then again if something breaks at the weekend, I would want to know about it on Monday morning, so that isn't a suitable solution.</p>
  <p>The ideal I guess would be to get one message to highlight an issue, but block all subsequent errors to cut down on noise in Slack. Would be great to hear what others are doing in this space</p>
  
Comments: CC BY-SA 4.0
Comments: 2024-05-29T06:28:17.170Z
Comments: what if you put the notifications in an SQS and you enable and disable processing that queue Mon-Fri 8am-6pm? That way you will lose nothing and the notifications may wait in the queue until Monday morning.
Comments: petrchpetr
Comments: CC BY-SA 4.0
Comments: 2024-05-29T14:15:06.750Z
Comments: This is good idea, working on a this just now, will report back...
Comments: e-on
Tags: <amazon-web-services><amazon-cloudwatch>
Title: How to optimize Cloudwatch alerts firing out-of-hours?
UserDisplayName: e-on
ThreadText: # How to optimize Cloudwatch alerts firing out-of-hours?
  
  (Asked by: username_0 on 2024-05-28)
  
  We have a range of AWS/NatGateway alerts set up to monitor the health of our networking. When the cloudwatch alerts fire, this goes to an SNS/Lambda, which then fires a message to Slack. One of the alerts is an ActiveConnectionCount, which fires if the nat gateway has zero connections. However, outside of regular office hours, it is really common for zero connections. What this leads to is dozens of messages sitting in Slack on Monday morning, and there is no problem to resolve.
  
  While this specific alert was an issue, it has led to a wider question: Has anyone found a good solution to manage cloudwatch alerting out of hours? I haven't found a way to set a time in cloudwatch to run the alerts, such as Mon-Fri 8am-6pm, i don't think it's possible. I could set a cron job to disable and enable alert actions which could just disable alerts outside office hours, but then again if something breaks at the weekend, I would want to know about it on Monday morning, so that isn't a suitable solution.
  
  The ideal I guess would be to get one message to highlight an issue, but block all subsequent errors to cut down on noise in Slack. Would be great to hear what others are doing in this space
  
  **Tags:** amazon-web-services, amazon-cloudwatch
  
  > **Comments:**
  > **username_1**: what if you put the notifications in an SQS and you enable and disable processing that queue Mon-Fri 8am-6pm? That way you will lose nothing and the notifications may wait in the queue until Monday morning.  
  > **username_0**: This is good idea, working on a this just now, will report back...  
  
  ---
  
  ## 0 answers
  
  