Minimizing 2am issues, and maintaining an on-call rotation, aren&#x27;t contradictory. There&#x27;s no substitute for having someone on call; but you can minimize the number of times they actually get called. This topic is near and dear to my heart.<p>You talk about testing - that&#x27;s one side of the coin; the other side is careful alert tuning, (a) to minimize false positives at 2 AM, (b) to catch incipient issues while it&#x27;s still business hours. (It&#x27;s useful to think of alerts as just another phase of QA - the one that occurs after you hit production. The sooner you notice a problem, the less damage it causes, to both your customers and your sleep schedule.)<p>At my workplace, we run a fairly complex system, but we&#x27;ve been able to keep nighttime pager incidents down to I think less than once per quarter, including false alarms. I can&#x27;t remember the last one. The QA effort isn&#x27;t overwhelming, either. See <a href="http:&#x2F;&#x2F;blog.scalyr.com&#x2F;2014&#x2F;08&#x2F;99-99-uptime-9-5-schedule&#x2F;" rel="nofollow">http:&#x2F;&#x2F;blog.scalyr.com&#x2F;2014&#x2F;08&#x2F;99-99-uptime-9-5-schedule&#x2F;</a> if you&#x27;re interested.