Why I've lost faith in p values
Here&#x27;s a more simple thought experiment that gets across the point of why p(null | significant effect) &#x2F;= p(significant effect | null), and why p-values are flawed as stated in the post.<p>Imagine a society where scientists are really, really bad at hypothesis generation. In fact, they&#x27;re so bad that they only test null hypothesis that are true. So in this hypothetical society, the null hypothesis in any scientific experiment ever done is true. But statistically using a p value of 0.05, we&#x27;ll still reject the null in 5% of experiments. And those experiments will then end up being published in scientific literature. But then this society&#x27;s scientific literature now only contains false results - literally all published scientific results are false.<p>Of course, in real life, we hope that our scientists have better intuition for what is in fact true - that is, we hope that the &quot;prior&quot; probability in Bayes&#x27; theorem, p(null), is not 1.