Answers: <p><strong>DTAP</strong></p>
  
  <p>As monitoring is part of production it should be part of DTAP as well. This implies monitoring in Development, Testing, Acceptance and Production as well. If somebody modifies a check in production and it will silence some check and if the team does not get reported that there are issues then the customer will be impacted and that could be a huge problem. In short, if monitoring is part of production it should be applied to all stages of DTAP as well.</p>
  
  <p><strong>Testing</strong></p>
  
  <p>If custom scripts are used in splunk and these are written in Python or some other language, then you should also apply unit tests and integration tests. In most monitoring systems, a 0, 1, 2 and 3 exit code are checked so this should also be covered in the tests. If a monitoring is written in bash, one could use BATS and Powershell can be tested as well with Pester. </p>
  
  <p>Why testing these scripts? Again, same reason as depicted in the DTAP section. Imagine that somebody breaks some monitoring script by introducing a typo and you do not get notified, then this could have a huge impact on the customer and also on the team. Imagine that you have to debug for a couple of days because some script is not working well, while that could have been prevented. So I advocate to apply unit tests, integration tests and even CI for this "simple" monitoring scripts.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-05-04T20:46:14.943Z
Answers: 2020-05-04T20:46:14.943Z
Answers: 030
Body: <h1>Background</h1>
  
  <p>Our <strong>code</strong> is written with:</p>
  
  <ol>
  <li>Unit tests</li>
  <li>End to end tests</li>
  <li>Code review</li>
  <li>Staging process</li>
  <li>Deployment process</li>
  </ol>
  
  <p>On the contrary, our <strong>alerts</strong> are just written and then modified occasionally manually. No quality process at all.</p>
  
  <p>This process is reasonable for simple threshold checks. However, our alerts are sometimes built on complicated queries. Sometimes composed of ~20 lines of a query.</p>
  
  <p>If we accidentally break an alert, it could expose us to production instability since we won't know if some logic or component breaks.</p>
  
  <h1>The question</h1>
  
  <p>Is there a recommended methodology for validating the quality of complicated alerts?</p>
  
  <h2>P.S.</h2>
  
  <p>We're using Splunk alerts</p>
  
Tags: <monitoring>
Title: Production grade methodology for alerts
UserDisplayName: Michael
ThreadText: # Production grade methodology for alerts
  
  (Asked by: username_0 on 2020-05-03)
  
  # Background
  
  Our **code** is written with:
  
  1. Unit tests
  2. End to end tests
  3. Code review
  4. Staging process
  5. Deployment process
  
  On the contrary, our **alerts** are just written and then modified occasionally manually. No quality process at all.
  
  This process is reasonable for simple threshold checks. However, our alerts are sometimes built on complicated queries. Sometimes composed of ~20 lines of a query.
  
  If we accidentally break an alert, it could expose us to production instability since we won't know if some logic or component breaks.
  
  # The question
  
  Is there a recommended methodology for validating the quality of complicated alerts?
  
  ## P.S.
  
  We're using Splunk alerts
  
  **Tags:** monitoring
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-04. Score: 0 | Accepted answer
  
  **DTAP**
  
  As monitoring is part of production it should be part of DTAP as well. This implies monitoring in Development, Testing, Acceptance and Production as well. If somebody modifies a check in production and it will silence some check and if the team does not get reported that there are issues then the customer will be impacted and that could be a huge problem. In short, if monitoring is part of production it should be applied to all stages of DTAP as well.
  
  **Testing**
  
  If custom scripts are used in splunk and these are written in Python or some other language, then you should also apply unit tests and integration tests. In most monitoring systems, a 0, 1, 2 and 3 exit code are checked so this should also be covered in the tests. If a monitoring is written in bash, one could use BATS and Powershell can be tested as well with Pester.
  
  Why testing these scripts? Again, same reason as depicted in the DTAP section. Imagine that somebody breaks some monitoring script by introducing a typo and you do not get notified, then this could have a huge impact on the customer and also on the team. Imagine that you have to debug for a couple of days because some script is not working well, while that could have been prevented. So I advocate to apply unit tests, integration tests and even CI for this "simple" monitoring scripts.
  
  
  ---
  
  