Answers: <p>We ended up creating a new PagerDuty service with a schedule attached to it.  That schedule only had anybody assigned for half of the day.  This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-09-02T12:18:20.000Z
Answers: 2019-09-02T12:18:20.000Z
Answers: chicks
Body: <h1>Initial Problem</h1>
  
  <p>We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps.  These are metrics that would be problematic if they happened during the day when customers are using the service, but at night there are absolutely no customers so there's no motivation by devs to make the nightly jobs less error-prone.  So now I want to make it so it doesn't alert every night.  Between New Relic and PagerDuty I would think I could figure that out, but so far I've come up with nothing practical.</p>
  
  <h1>New Relic</h1>
  
  <p>I haven't spent tons of time with New Relic so I was learning as I tried adjusting things, but based on <a href="https://discuss.newrelic.com/t/only-send-alerts-during-a-specifc-time-window-each-day/27560/30" rel="nofollow noreferrer">this thread on New Relic's site</a> it seems clear that time-boxing alerts is not a feature they current support or will in the forseeable future.  I've dealt with using history as a predictor for future results using a few technologies and it is a mess that would not be a reasonable replacement for the simple alerts that are in place currently.  Going down that path would create more noisy alerts without detecting any more problems than we already find.</p>
  
  <p>[ It doesn't seem very DevOps-friendly to me to ignore feature requests for simple features that would solve lots of people's problems and only offer complicated quagmires as an alternative.  I'm sure they're all jazzed about 24-7 cloud stuff, but ignoring the market of people who have prehistoric needs is sad.  My audience is literally captive and they are going to stay that way until we stop putting people in prison.  This use case will not go away regardless of how much cloud pixie dust you sprinkle on it. ]</p>
  
  <h1>PagerDuty</h1>
  
  <p>I've used PagerDuty merrily for years and I was sure there would be some way to handle this after the alert left New Relic, but:</p>
  
  <ul>
  <li><a href="https://support.pagerduty.com/docs/schedules" rel="nofollow noreferrer">Schedules</a> apply to people and come after an alert has been through a Service.  If you could add a Service to a Schedule we could hack this, but no, that's not there.</li>
  <li><a href="https://support.pagerduty.com/docs/event-management#section-suppression-and-event-rules" rel="nofollow noreferrer">Event Rule-based Suppression</a> doesn't include time-of-day conditions.</li>
  <li><a href="https://support.pagerduty.com/docs/maintenance-windows" rel="nofollow noreferrer">Maintenance Windows</a> might work, but I was really hoping this wouldn't turn into another cron job I have to keep an eye on.</li>
  </ul>
  
  <h1>The Question</h1>
  
  <p>Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-09-02T10:50:33.037Z
Comments: How did you end up solving the problem? Maintenance windows?
Comments: Kaushik Thirthappa
Tags: <monitoring><pagerduty>
Title: Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?
UserDisplayName: chicks
ThreadText: # Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?
  
  (Asked by: username_0 on 2019-01-09)
  
  # Initial Problem
  
  We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps. These are metrics that would be problematic if they happened during the day when customers are using the service, but at night there are absolutely no customers so there's no motivation by devs to make the nightly jobs less error-prone. So now I want to make it so it doesn't alert every night. Between New Relic and PagerDuty I would think I could figure that out, but so far I've come up with nothing practical.
  
  # New Relic
  
  I haven't spent tons of time with New Relic so I was learning as I tried adjusting things, but based on [this thread on New Relic's site](https://discuss.newrelic.com/t/only-send-alerts-during-a-specifc-time-window-each-day/27560/30) it seems clear that time-boxing alerts is not a feature they current support or will in the forseeable future. I've dealt with using history as a predictor for future results using a few technologies and it is a mess that would not be a reasonable replacement for the simple alerts that are in place currently. Going down that path would create more noisy alerts without detecting any more problems than we already find.
  
  [ It doesn't seem very DevOps-friendly to me to ignore feature requests for simple features that would solve lots of people's problems and only offer complicated quagmires as an alternative. I'm sure they're all jazzed about 24-7 cloud stuff, but ignoring the market of people who have prehistoric needs is sad. My audience is literally captive and they are going to stay that way until we stop putting people in prison. This use case will not go away regardless of how much cloud pixie dust you sprinkle on it. ]
  
  # PagerDuty
  
  I've used PagerDuty merrily for years and I was sure there would be some way to handle this after the alert left New Relic, but:
  
  * [Schedules](https://support.pagerduty.com/docs/schedules) apply to people and come after an alert has been through a Service. If you could add a Service to a Schedule we could hack this, but no, that's not there.
  * [Event Rule-based Suppression](https://support.pagerduty.com/docs/event-management#section-suppression-and-event-rules) doesn't include time-of-day conditions.
  * [Maintenance Windows](https://support.pagerduty.com/docs/maintenance-windows) might work, but I was really hoping this wouldn't turn into another cron job I have to keep an eye on.
  
  # The Question
  
  Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?
  
  **Tags:** monitoring, pagerduty
  
  > **Comments:**
  > **username_1**: How did you end up solving the problem? Maintenance windows?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-09-02. Score: 2 | Accepted answer
  
  We ended up creating a new PagerDuty service with a schedule attached to it. That schedule only had anybody assigned for half of the day. This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.
  
  
  ---
  
  