&gt; To make things worse all our usual monitoring hadn&#x27;t caught the problem<p>I suspect this is because most monitoring focuses on average instead of percentiles or maximum. Outliers are the most interesting data points.  There is an almost cult-like following of the average as a KPI in monitoring tools.