Reproducibility trial: Biologists get different results from same data sets
The issue here is the same across all &#x27;science&#x27; which is unable to effectively control causes, and which studies non-linear phenomena (or, more simply, cases where the distribution of means isnt defined, or is far away from normal).<p>The issue is: you can&#x27;t do stats. The reality is it is highly likely that all data here is consistent with any effect size you care to choose.<p>What this study does, nevertheless, is quantify the &#x27;model risk&#x27; ie., the &quot;p-value of the method itself&quot; -- how often does research provide reliable effect sizes? Rarely. So the research methods themselves have a huge &#x27;epistemic discounting&#x27; associated with them.<p>But we can go much further than this -- recurse: what is the risk of using this methodology to assess methodology? Actually, it&#x27;s fairly high. This isn&#x27;t a very reliable method for judging research methods. And so on.<p>What we find here is that in the vast majority of cases, where we&#x27;d care to research anything, recursing risks lead to an essential insurmountable error: measurement error, model error, methodology error, research-assumptions-error, etc. There isnt enough data to resolve between competing possibilities at the outer-most stage of error (often you&#x27;d need much more data than practically possible to collect).<p>What do we do with this? I&#x27;d say, mostly, throw away stats here. Do case analyses, risk-based recommendations, even broadly philosophical analysis.<p>The alterantive is just staring at tea-leaves.