Sort of covered:<p>"Ideally, you've designed your system such that real-time human intervention is never required; every failure should be handled automatically, with repairs needing human hands neatly enqueued for processing during work hours."<p>My takeaway: Above a certain scale, one needs to stop responding to individual failures, and start responding to statistics.