Monitoring demystified: A guide for logging, tracing, metrics
A few things I have learnt along the way:Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late.Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx and make a graph, collate and push the metrics directly)Raw metrics are pretty useless. They need to be manipulated into buisness goals: service x is producing 3% 5xx errors vs % of visitors unable to perform action xAlerts must be actionable.Alerts rules must be based on sensible clear cut rules: service x's response time is breeching its SLA not service x's response time is double its average for this time in may.