Car alarms and smoke alarms: tradeoff between sensitivity and specificity (2012)
I do like how the author presents the case for how damaging false-positives can be in SRE monitoring. But, FYI, it can get worse if these monitors are hooked to self-actuating feedback loops! I recently wrote about a production incident on the Heii On-Call blog, in the context of witnessing how Kubernetes liveness probes and CPU limits worked together to create a self-reinforcing CrashLoopBackOff. [1] Partially because the liveness probe thresholds (timeoutSeconds and failureThreshold fields) were too aggressive.<p>We have a similar message about setting monitoring thresholds in our documentation [2] because users have to explicitly specify a downtime timeout before they’re alerted about their website &#x2F; API endpoint &#x2F; cron job being down. The timeout &#x2F; &quot;grace period&quot; is necessary because in many cases a failure is some transient network glitch which will fix itself before a human is alerted.<p>If you make the timeout too short, you’ll get lots of false positive alerts, and as the article says, your on-call engineers will be overwhelmed or just start ignoring the alerts.<p>If you make the timeout too long, it just takes that many minutes of downtime longer before you find out about it.<p>It may sound counterintuitive, but the latter is usually preferable. :)<p>[1] <a href="https:&#x2F;&#x2F;heiioncall.com&#x2F;blog&#x2F;kubernetes-liveness-probes-and-cpu-limits-risks-self-reinforcing-crashloopbackoff" rel="nofollow">https:&#x2F;&#x2F;heiioncall.com&#x2F;blog&#x2F;kubernetes-liveness-probes-and-c...</a><p>[2] <a href="https:&#x2F;&#x2F;heiioncall.com&#x2F;docs" rel="nofollow">https:&#x2F;&#x2F;heiioncall.com&#x2F;docs</a>