Answers: <p>Consider a service using the alerting rule from the book:</p>
  
  <p><code>job:slo_errors_per_request:ratio_rate1h{job="myjob"} &gt; 36 * 0.001</code></p>
  
  <p>The burn rate is the rate at which we are exhausting our error budget. A burn rate of 1 means that we'll have exactly exhausted our error budget at the end of our reporting period. A burn rate of 2 means that we'll have exhausted our budget half way through our reporting period.</p>
  
  <p>For a 99.9% SLO, an error rate of 0.1% constitutes a burn rate of 1. An error rate of 1% means a burn rate of 10, and a 100% error rate means a burn rate of 1000</p>
  
  <p>If we want to alert when we've exhausted 5% of our 30 day error budget in 1 hour, then that means we are burning error budget at a rate of 36, which translates to an error rate of 3.6% for a 99.9% SLO.</p>
  
  <p>Imagine a service with a constant rate of requests, that starts serving 100% errors at t=0 minutes</p>
  
  <p>Here our SLI counts the total number of "events".</p>
  
  <p>SLIs written in terms of some kind of discrete events, where an event can either be good or bad, have many nice properties: They range from 0-100%, where 0 is "nothing is working" and 100% is "nothing is broken"; I find this quite intuitive. By making all of your SLIs behave the same, you simplify tooling, alerting rules, and understanding.</p>
  
  <p>With a request-based system, events are usually individual requests. So 0 errors, 1 request in minute at t=-2. I used 1 request per minute for simplicity. The result is the same if you have 1 request per minute or 1 million; it's the ratio of errors to total requests that matters.</p>
  
  <p>t=-2, errors=0, total=1<br>
  t=-1, errors=0, total=1<br>
  t=0, errors=1, total=1<br>
  t=1, errors=1, total=1<br>
  ...</p>
  
  <p>If our monitoring system evaluates its rules every minute, we get:</p>
  
  <p>at t=-1, our 1h average error rate is 0<br>
  at t=0, the 1h error rate is 1/60 = 1.666%<br>
  at t=1, the 1h error rate is 2/60 = 3.333%<br>
  at t=2, the 1h error rate is 3/60 = 5%</p>
  
  <p>Our alert fires 2 minutes after the event started.</p>
  
  <p>Let's assume our incident continues until t=60.</p>
  
  <p>at t=60, our 1h error rate is 100%: a burn rate of 1000, for a 99.9% SLO<br>
  at t=61, the 1h error rate drops to 59/60 = 98.333%<br>
  at t=62, the 1h error rate is 58/60 = 96.666%<br>
  at t=118, the 1h error rate is 2/60 = 3.333%  </p>
  
  <p>At this point, our 1h error rate is below the alerting threshold, and so the alert is no longer in the firing state, 58 minutes after the end of the event.</p>
  
  <p>at t=120, our 1h error rate finally drops back to 0%, as no errors have been observed in the last hour.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-06-07T10:53:41.890Z
Answers: In `t=-2, errors=0, total=1`, what does `total` refer to?
Answers: user15207
Answers: CC BY-SA 4.0
Answers: 2019-06-11T08:09:24.897Z
Answers: total is the total number of requests; I have edited the answer to make this clearer.
Answers: Steven Thurgood
Answers: CC BY-SA 4.0
Answers: 2019-06-04T14:31:46.313Z
Answers: 2019-06-11T08:13:20.833Z
Answers: 2019-06-11T08:13:20.833Z
Answers: Steven Thurgood
Body: <p>In <a href="https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r" rel="nofollow noreferrer">https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r</a> I don't understand how the reset time can be 58 min when the alerting window size is one hour.</p>
  
  <p>How can the reset time be shorter than the alerting window size?</p>
  
Tags: <monitoring><cloud><sre>
Title: Reset time vs alerting window size in alerting strategies for SLOs
UserDisplayName: user15207
ThreadText: # Reset time vs alerting window size in alerting strategies for SLOs
  
  (Asked by: username_0 on 2019-05-30)
  
  In <https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r> I don't understand how the reset time can be 58 min when the alerting window size is one hour.
  
  How can the reset time be shorter than the alerting window size?
  
  **Tags:** monitoring, cloud, sre
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-04. Score: 1 | Accepted answer
  
  Consider a service using the alerting rule from the book:
  
  `job:slo_errors_per_request:ratio_rate1h{job="myjob"} > 36 * 0.001`
  
  The burn rate is the rate at which we are exhausting our error budget. A burn rate of 1 means that we'll have exactly exhausted our error budget at the end of our reporting period. A burn rate of 2 means that we'll have exhausted our budget half way through our reporting period.
  
  For a 99.9% SLO, an error rate of 0.1% constitutes a burn rate of 1. An error rate of 1% means a burn rate of 10, and a 100% error rate means a burn rate of 1000
  
  If we want to alert when we've exhausted 5% of our 30 day error budget in 1 hour, then that means we are burning error budget at a rate of 36, which translates to an error rate of 3.6% for a 99.9% SLO.
  
  Imagine a service with a constant rate of requests, that starts serving 100% errors at t=0 minutes
  
  Here our SLI counts the total number of "events".
  
  SLIs written in terms of some kind of discrete events, where an event can either be good or bad, have many nice properties: They range from 0-100%, where 0 is "nothing is working" and 100% is "nothing is broken"; I find this quite intuitive. By making all of your SLIs behave the same, you simplify tooling, alerting rules, and understanding.
  
  With a request-based system, events are usually individual requests. So 0 errors, 1 request in minute at t=-2. I used 1 request per minute for simplicity. The result is the same if you have 1 request per minute or 1 million; it's the ratio of errors to total requests that matters.
  
  t=-2, errors=0, total=1  
  t=-1, errors=0, total=1  
  t=0, errors=1, total=1  
  t=1, errors=1, total=1  
  ...
  
  If our monitoring system evaluates its rules every minute, we get:
  
  at t=-1, our 1h average error rate is 0  
  at t=0, the 1h error rate is 1/60 = 1.666%  
  at t=1, the 1h error rate is 2/60 = 3.333%  
  at t=2, the 1h error rate is 3/60 = 5%
  
  Our alert fires 2 minutes after the event started.
  
  Let's assume our incident continues until t=60.
  
  at t=60, our 1h error rate is 100%: a burn rate of 1000, for a 99.9% SLO  
  at t=61, the 1h error rate drops to 59/60 = 98.333%  
  at t=62, the 1h error rate is 58/60 = 96.666%  
  at t=118, the 1h error rate is 2/60 = 3.333%
  
  At this point, our 1h error rate is below the alerting threshold, and so the alert is no longer in the firing state, 58 minutes after the end of the event.
  
  at t=120, our 1h error rate finally drops back to 0%, as no errors have been observed in the last hour.
  
  > **Comments:**
  > **username_0**: In `t=-2, errors=0, total=1`, what does `total` refer to?  
  > **username_1**: total is the total number of requests; I have edited the answer to make this clearer.  
  
  ---
  
  