Coverage is not strongly correlated with test suite effectiveness
<p><pre><code>    In other words, more tests do find more bugs, but it&#x27;s the number of tests and not their code coverage that has most of the predictive value. It&#x27;s a surprising result, so if you&#x27;ll excuse me, I have a couple of lecture slides on software testing I need to revise
</code></pre>
Is it just me or was this _not_ surprising at all?<p>I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other people about this but to me it&#x27;s a well known fact. There are so many ways this can go wrong.<p>I mean it&#x27;s so easy to give the one counter example needed to break the myth of 100% test coverage being good for much: Well you executed the branch&#x2F;line at least once with one potential input. Was it an edge case input or a happy path input?<p>More tests than is needed for &quot;100% coverage&quot; means that you actually executed some lines multiple times, hopefully with not just 10 happy path scenarios but with 1 happy path and 9 edge cases. Now remove the happy path scenarios for trivial code and also the edge case scenarios for trivial code and your coverage might only be 80% but you have the same actual test suite effectiveness. When you keep adding tests, add more to the 9 edge cases, staying with the same coverage but make the suite more robust.<p>Of course 20% is better than 0% and 50% is better than 20%. Somewhere between 50 and 100 is an optimal point. For lack of a good estimate, let&#x27;s go with the old 80&#x2F;20 rule. Something around 80% test coverage is what you can use to make a tool fail the build. If you pass the 80% mark you _might_ have a good test suite but it doesn&#x27;t mean you stop there. It&#x27;s just a reminder that you might have missed adding tests when that thing triggers a red build but don&#x27;t use it to mean that you actually have a good test suite.