Operating a large distributed system in a reliable way: practices I learned
Don't really agree that this list could have come about through discussions with engineers at Google, Facebook, etc.  The more computers you have the less important it becomes to monitor junk like CPU and memory utilization of individual machines. Host-level CPU usage alerting can't possibly be a "must-have" if there are extremely large distributed systems operating without it.If you've designed software where the whole service can degrade based on the CPU consumption of a single machine, that right there is your problem and no amount of alerting can help you.