Robust research needs many lines of evidence
This paper really sounds like it advocates the status quo of attempting to take the &quot;sum of the evidence&quot;, which usually leads to the generation of many different weak (low sample size, etc) lines of evidence. They don&#x27;t seem to get that what made science so successful was its ability to generate simpler models of phenomenon. To do this you need to come up with (most often quantitative) theories that can synthesize various lines of evidence.<p>The practice of &quot;counting heads&quot; seen in fields like medicine is totally inappropriate (eg 2 out of 3 results support the theory so it looks good). The correct thing to do is more like what is seen in physics.<p>Eg, take the idea that &quot;cancer is due to mutations that accumulate as cells divide, therefore cancer rates should be related to number of divisions and mutation rate.&quot; The status quo is to do something like:<p><pre><code>  1) &quot;cancer is positively correlated with number of divisions&quot; 
  2) &quot;cancer is positively correlated with mutation rate&quot;
  3) &quot;cancer is positively correlated with number of accumulated mutations&quot;

</code></pre>
Instead, we should say &quot;this means the probability of accumulating at least <i>n</i> mutations after <i>d</i> divisions given mutation rate <i>r</i> should follow a law similar to&quot;:<p><pre><code>   p(x &gt;= n) = (1 - (1 - r)^d)^n
</code></pre>
The two approaches are similar in that you measure mutation rate, number of divisions, accumulated mutations and frequency of cancer cells. The difference is that for the latter approach all the lines of evidence combine in a <i>nonlinear</i> fashion.<p>This method allows us to subsume a myriad of factors such as genetic disposition, exposure to toxins, etc to be incorporated via a single value (mutation rate). So rather than an ever expanding apparent complexity as more factors are incorporated, there is an accumulation of knowledge as we estimate the model parameters more and more accurately and either reject, modify, or keep our models of the phenomenon.