Suspicious Discontinuities
Unfortunately I can't find the reference any more, but there was this great presentation by someone explaining how in computer systems developers will "optimise for the requirement", and ignore anything outside of it.The example was latency. If the programmers were told to achieve less than 1 ms for 99% of the requests -- then sure enough -- the 1% of requests would have sky-high latencies of multiple seconds.If told they needed to achieve 1 ms for 99% and 100 ms for 99.99%, then -- you guessed it -- the worst 0.01% would be tens of seconds or even minutes.Inevitably, there would be a visible discontinuity in the latency histogram just above whatever the official business requirement was.It's a difficult thing to fix, because no matter where you set your threshold, unless it's 100%, you won't meet it. And even 100% is just a fiction, because you'd need an infinite number of tests to achieve it.