Paper author here.  We measured &quot;effectiveness&quot; as the mutation score.  We showed in a separate paper that the mutation score is a good way to measure a test suite&#x27;s ability to detect real faults.  Of course, using the mutation score instead of the real fault detection rate does add a layer of indirection, but it&#x27;s a lot more practical for a large empirical study (i.e., can be automated), so there&#x27;s a bit of a tradeoff there.  (The other paper is also on my website if you want to read it: <a href="http:&#x2F;&#x2F;www.linozemtseva.com&#x2F;research&#x2F;2014&#x2F;fse&#x2F;mutant_validity&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.linozemtseva.com&#x2F;research&#x2F;2014&#x2F;fse&#x2F;mutant_validit...</a>)<p>What the graphs show is that effectiveness rises with suite size, which is expected: more tests catch more bugs.  We also see that effectiveness rises with coverage, which seems intuitive: you can&#x27;t catch bugs in code you never run.  But when you graph effectiveness against coverage for suites that are all the <i>same</i> size, the correlation drops significantly, in some cases to 0.  Here&#x27;s an analogy: the number of PhD students that graduate in the US is highly correlated with the amount of profit generated by arcades in the US, but there&#x27;s no causal relationship between them.  They probably both depend on the size of the population.  Similarly, coverage and effectiveness are correlated because they both depend on the size of the suite, but we can&#x27;t say that there&#x27;s a causal relationship.  In other words, saying that a suite will catch a lot of bugs because it has high coverage is like saying that a lot of PhD students will graduate because arcades turned a good profit this year.<p>Your third point is a really good question.  It&#x27;s true that developers don&#x27;t write tests randomly, so our method of making new suites by picking random test cases isn&#x27;t quite realistic.  What impact that would have on the results, I&#x27;m not sure.  That&#x27;s something I&#x27;d like to look into in the future.