Coverage Is Not Strongly Correlated with Test Suite Effectiveness (2014)
I have a few objections to the evidence presented.<p>Keeping in mind that I don&#x27;t disagree with the statement. Test coverage is an objective metric, and test Effectiveness is a... what is it again? How many bugs you&#x27;ll find with it? Obviously the two are separate concepts.<p>This is my first objection: The paper seems to say that Mutation Testing is test effectiveness, but Mutation testing is merely another metric. They cite other papers, but papers that attempt to demonstrate that this metric is correlated with test &quot;effectiveness&quot;.<p>Metric against metric, is that meaningful?<p>She presents graphs of results that seem to demonstrate linear correlation between test suite size&#x2F;test suite coverage&#x2F;and mutation testing (called &quot;effectiveness&quot;) This is addressed later, &quot;isn&#x27;t this what we would expect?&quot; yah! And the explanation for why it&#x27;s unexpected sailed fully over my head. (I admit it! I&#x27;m dumb.)<p>finally, many test suites are generated on the presumption of achieving code coverage, is this test valid without also having test suites that were made without that goal in mind? Could such a suite exist?<p>so summary of my objections<p>* is mutation testing a meaningful measure of effectiveness?<p>* can you measure one metric against another another, get a linear relationship, and conclude any meaningful differences?<p>* does the presence of code coverage as a target spoil the conclusion?<p>I&#x27;d love to hear input on this.