Instrumentation: The First Four Things You Measure
&gt; A histogram of the duration it took to serve a response to a request, also labelled by successes or errors.<p>I recommend against this, rather have one overall duration metric and another metric tracking a count of failures.<p>The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed.<p>The more information you put on a dashboard, the more chance someone will miss a subtlety like this in the interpretation. Particularly if debugging distributed systems isn&#x27;t their forte, or they&#x27;ve been woken up in the middle of the night by a page.<p>This guide only covers what I&#x27;d consider online serving systems, I&#x27;d suggest a look at the Prometheus instrumentation guidelines on what sort of things to monitor for other types of systems: <a href="https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;practices&#x2F;instrumentation&#x2F;" rel="nofollow">https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;practices&#x2F;instrumentation&#x2F;</a>