Answers: <p>My current understanding is that it is not possible to avoid false positives. At least not if traffic comes from different time zones around the world (as in the context of this question).</p>
  
  <p>AWS operates on UTC times, but this is irrelevant for the anomaly detection. It does play a role if you set timers, but it is not taking into account for the predictions of AWS. In other words, switching to UTC will not make false-positives less likely.</p>
  
  <p>When thinking about it, it is hard to avoid false-positives. If a  time-shift in Europe or the US happens, you could take that  into account and predict the new traffic. But that requires extra domain knowledge. The AWS predictor only sees a  generic time series.</p>
  
  <p>There are other events such as Christmas, which will impact the expected traffic. Again, with extra knowledge you could improve predictions. But the problem is too difficult for the AWS predictor, as it does not have that domain knowledge.</p>
  
  <p>In our concrete case, we still kept the alerts, as they do have value even with the known false-positives. Humans with knowledge about the underlying system, can reject the false-positive. It is not ideal, but it is a trade-off that we are willing to make.</p>
  
  <p>Note that other types of graphs are not susceptible for these types of problems. For example, error rates instead of total number of errors. It is not always possible, but if you can use rates, the system will be more stable in my experience.
  AWS allows to set alerts on math expressions such as <code>errors / requests</code>. The advantage is that normalized rates tend to be agnostic of external events.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-12-20T17:51:59.943Z
Answers: 2019-12-20T17:51:59.943Z
Answers: Philipp Claßen
Body: <p>For monitoring, I set up CloudWatch alerts based on <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html" rel="nofollow noreferrer">CloudWatch anomaly detection</a>. Overall, they work quite nicely, but they got confused when the clock gets shifted (summer timer to winter time).</p>
  
  <p>We have now recurring false-positives even though the recent time shift in the US and in Europe were weeks ago. I assume CloudWatch will eventually understand it, but then again, the clock shifts have on a regular basis, which will create some noisy.</p>
  
  <p>Unfortunately, I did not find much in the documentation, only that CloudWatch provides to set the <code>Time Zone Format</code>. Before I leaved it empty, but now I set it to UTC.</p>
  
  <p><strong>Questions:</strong></p>
  
  <ul>
  <li>What is exactly the effect of setting the time zone? Does it have an impact on the prediction of the anomaly detector, or is it only for visualization and for other configurations such as excluding specified time periods?</li>
  <li>Our user base is all over the world, but mostly in the US and in Europe. As this scenario is not too uncommon, I wonder if asking for one one time zone is even practical. What should be used? UTC? Or pick the time zone with the most users?</li>
  </ul>
  
  <p>I understand that it difficult to avoid false-positives completely, especially, in the first or second week after a clock shift happened. Still, I wonder how to correctly setup up the anomaly detection model, so its prediction will be most accurate.</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-12-20T11:10:49.777Z
Comments: Could you indicate whether the issues have been solved after two weeks?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2019-12-20T16:23:58.250Z
Comments: @030 Nothing new from my side. Still open.
Comments: Philipp Claßen
Comments: CC BY-SA 4.0
Comments: 2019-12-20T16:27:32.723Z
Comments: https://www.reddit.com/r/aws/comments/c9e55t/having_time_issues_with_cloudwatch_event_rules/eswusqc/
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2019-12-20T16:36:16.330Z
Comments: @030 Thanks for the link. My current understanding is also that there is not much to be done (regarding monitoring). The anomaly detector from AWS will eventually pick up the new pattern but it takes a while. With traffic from different regions of the world, I'm not even sure if it is possible for a smarter algorithm to detect it. I also expect that events like Christmas (where the traffic will drop significantly) will lead to waves of false positives (at least in the first week after Christmas when traffic comes back).
Comments: Philipp Claßen
Comments: CC BY-SA 4.0
Comments: 2019-12-20T16:38:22.400Z
Comments: Could you post as an answer?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2019-12-20T17:53:00.220Z
Comments: @030 Done. Posted an answer with my current understanding.
Comments: Philipp Claßen
Comments: CC BY-SA 4.0
Comments: 2019-12-20T18:21:50.640Z
Comments: Thanks. I will upvote it tomorrow `Daily vote limit reached; vote again in 5 hours.`
Comments: 030
Tags: <amazon-web-services><amazon-cloudwatch>
Title: Mitigate false positives due to daylight saving time in CloudWatch anomaly dectector
UserDisplayName: Philipp Claßen
ThreadText: # Mitigate false positives due to daylight saving time in CloudWatch anomaly dectector
  
  (Asked by: username_0 on 2019-11-25)
  
  For monitoring, I set up CloudWatch alerts based on [CloudWatch anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html). Overall, they work quite nicely, but they got confused when the clock gets shifted (summer timer to winter time).
  
  We have now recurring false-positives even though the recent time shift in the US and in Europe were weeks ago. I assume CloudWatch will eventually understand it, but then again, the clock shifts have on a regular basis, which will create some noisy.
  
  Unfortunately, I did not find much in the documentation, only that CloudWatch provides to set the `Time Zone Format`. Before I leaved it empty, but now I set it to UTC.
  
  **Questions:**
  
  * What is exactly the effect of setting the time zone? Does it have an impact on the prediction of the anomaly detector, or is it only for visualization and for other configurations such as excluding specified time periods?
  * Our user base is all over the world, but mostly in the US and in Europe. As this scenario is not too uncommon, I wonder if asking for one one time zone is even practical. What should be used? UTC? Or pick the time zone with the most users?
  
  I understand that it difficult to avoid false-positives completely, especially, in the first or second week after a clock shift happened. Still, I wonder how to correctly setup up the anomaly detection model, so its prediction will be most accurate.
  
  **Tags:** amazon-web-services, amazon-cloudwatch
  
  > **Comments:**
  > **username_1**: Could you indicate whether the issues have been solved after two weeks?  
  > **username_0**: @username_1 Nothing new from my side. Still open.  
  > **username_1**: https://www.reddit.com/r/aws/comments/c9e55t/having_time_issues_with_cloudwatch_event_rules/eswusqc/  
  > **username_0**: @username_1 Thanks for the link. My current understanding is also that there is not much to be done (regarding monitoring). The anomaly detector from AWS will eventually pick up the new pattern but it takes a while. With traffic from different regions of the world, I'm not even sure if it is possible for a smarter algorithm to detect it. I also expect that events like Christmas (where the traffic will drop significantly) will lead to waves of false positives (at least in the first week after Christmas when traffic comes back).  
  > **username_1**: Could you post as an answer?  
  > **username_0**: @username_1 Done. Posted an answer with my current understanding.  
  > **username_1**: Thanks. I will upvote it tomorrow `Daily vote limit reached; vote again in 5 hours.`  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-12-20. Score: 1 | Accepted answer
  
  My current understanding is that it is not possible to avoid false positives. At least not if traffic comes from different time zones around the world (as in the context of this question).
  
  AWS operates on UTC times, but this is irrelevant for the anomaly detection. It does play a role if you set timers, but it is not taking into account for the predictions of AWS. In other words, switching to UTC will not make false-positives less likely.
  
  When thinking about it, it is hard to avoid false-positives. If a time-shift in Europe or the US happens, you could take that into account and predict the new traffic. But that requires extra domain knowledge. The AWS predictor only sees a generic time series.
  
  There are other events such as Christmas, which will impact the expected traffic. Again, with extra knowledge you could improve predictions. But the problem is too difficult for the AWS predictor, as it does not have that domain knowledge.
  
  In our concrete case, we still kept the alerts, as they do have value even with the known false-positives. Humans with knowledge about the underlying system, can reject the false-positive. It is not ideal, but it is a trade-off that we are willing to make.
  
  Note that other types of graphs are not susceptible for these types of problems. For example, error rates instead of total number of errors. It is not always possible, but if you can use rates, the system will be more stable in my experience.
  AWS allows to set alerts on math expressions such as `errors / requests`. The advantage is that normalized rates tend to be agnostic of external events.
  
  
  ---
  
  