Effect size is significantly more important than statistical significance
Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on &quot;significance&quot;. But I&#x27;m not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I&#x27;m interested in studies that reveal robust results that are reliable enough to be built on to produce other results. Results of 3% variations with p=0.046 aren&#x27;t. They&#x27;re dead ends, because you can&#x27;t put very many of those into the foundations of future papers before the probability of one of your foundations being incorrect is too large.<p>To the extent that those are hard to come by... Yeah! They are! Science is hard. Nobody promised this would be easy. Science <i>shouldn&#x27;t</i> be something where labs are cranking out easy 3%&#x2F;p=0.046 papers all the time just to keep funding. It&#x27;s just a waste of money and time of our smartest people. It <i>should</i> be harder than it is now.<p>Too many proposals are obviously only going to be capable of turning up that result (insufficient statistical power is often obvious right in the proposal, if you take the time to work the math). I&#x27;d rather see more wood behind fewer arrows, and see fewer proposals chasing much more statistical power, than the chaff of garbage we get now.<p>If I were King of Science, or at least, editor of a prestigious journal, I&#x27;d want to put word out that I&#x27;m looking for papers with at least one of some sort of <i>significant</i> effect, or a p value of something like p = 0.0001. Yeah. That&#x27;s a high bar. I know. That&#x27;s the point.<p>&quot;But jerf, isn&#x27;t it still valuable to map out all the little things like that?&quot; No, it really isn&#x27;t. We already have every reason in the world to believe the world is <i>drenched</i> in 1%&#x2F;p=0.05 effects. &quot;Everything&#x27;s correlated to everything&quot;, so that&#x27;s not some sort of amazing find, it&#x27;s the totally expected output of living in our reality. Really, this sort of stuff is still just <i>below the noise floor</i>. Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor, not spending billions of dollars doing the equivalent of listening to our spirit guides communicate to us over white noise from the radio.