Suspicious Discontinuities
Unfortunately I can't find the reference any more, but there was this great presentation by someone explaining how in computer systems developers will "optimise for the requirement", and ignore anything outside of it.<p>The example was latency. If the programmers were told to achieve less than 1 ms for 99% of the requests -- then sure enough -- the 1% of requests would have sky-high latencies of multiple seconds.<p>If told they needed to achieve 1 ms for 99% and 100 ms for 99.99%, then -- you guessed it -- the worst 0.01% would be tens of seconds or even minutes.<p>Inevitably, there would be a visible discontinuity in the latency histogram just above whatever the official business requirement was.<p>It's a difficult thing to fix, because no matter where you set your threshold, unless it's 100%, you won't meet it. And even 100% is just a fiction, because you'd need an infinite number of tests to achieve it.