Bayesian criticism of frequentist p-value I recently started reading the Bayesian criticism of the p-value and it seems that there is a lot of discussion around the fact that a frequentist approach is not that good when the Null Hypothesis is true.
For instance in this paper the authors write that "p-values overstate the evidence against the null [...] this does not have to do with type-I or type-II errors; it is an “independent” property of p-value."
To illustrate this point, the authors show that when the null is true, the p-value has a uniform distribution.
What I do not get is that even when the null is true, a frequentist approach, thanks to the Central Limit Theorem, is still able to construct confidence intervals that includes 0 (non-significance) at the appropriate $\alpha$ level.
I do not get why the fact that the p-value is uniform when the null is true shows that a frequentist approach is biased. And what does it mean "independent property of p-value"?

library(tidyverse)
library(broom)

n=1000
x = rnorm(n,100,30)
d = 0
y = x*d + rnorm(n,0,20)
df = data.frame(y,x)
plot(x,y)
abline(lm(y~x), col = 'red')

r = replicate(1000, sample_n(df, size = 50), simplify = F)
m = r %>% map(~ lm(y~x,data = .)) %>% map(tidy)

# Central Limit Theorem
bind_rows(.id = 'sample', m) %>% filter(term =='x') %>% ggplot(aes(estimate)) + facet_grid(~term) + geom_histogram()

s = bind_rows(.id = 'sample', m) %>% filter(term =='x')
s$false_positive = ifelse(s$p.value < 0.05, 1, 0)
prop.table(table(s$false_positive))

# uniform
hist(s$p.value, breaks = 50)


I think you’re conflating two different arguments against p-values. Let me spell them out.

*

*By definition, p is distributed uniformly under the null hypothesis (or as uniformly as possible in discrete settings). So p isn’t going to be a useful measure of evidence in favour of the null when it’s true. Evidence in favour of the null isn’t going to accumulate as we collect data, as p just makes a random walk bounded by 0 and 1. I don’t see where the your source mentions bias in this context.


*There exist mathematical theorems showing that p is much less than the posterior of the null for any choices of priors for parameters in the alternative hypothesis. This is what’s meant by overstating the evidence. For example, p = 0.05 under some assumptions corresponds to a null hypothesis that is at least about 30% plausible.
Let me note that p isn’t necessary at all in frequentist inference, and it’s use isn’t supported by any coherent framework. Indeed, there are really two kinds of frequentism here.

*

*Fisher, in which p is used and used as a measure of evidence. It is this one that the above arguments attack


*Neyman-Pearson, in which if p is used at all it’s used only to define a rejection region. The connection - or lack thereof - between p and evidence is neither here nor there.
Lastly, let me clarify confusion about the connection between NHST and proof by contradiction. In NHST, when faced with a small p-value, we face the dichotomy

*

*we observed a small p and the null was true or

*we observed a small p and the null was false

which evidently reduces to nothing more than the fact that we observed small p. It gets us nowhere deductively. Proof by contradiction on the other hand obviously does allow us to deductively prove non-trivial things. What required here isn’t deductive logic - like proof by contradiction - but inductive logic.
