ANOVA vs likelihood ratio test (different result) I found the likelihood ratio test has a very similar setup as ANOVA: we are essentially testing if adding an additional variable would significantly increase the fitness. So I run both statistical tests on a simple dataset.
> y = c(1,2,3,5,4,3)
> x = c("control", "control", "control", "treatment", "treatment","treatment")

# compare a more complicated model with a simple model, using anova
> anova(lm(y~1), lm(y~x))

## Analysis of Variance Table
##
## Model 1: y ~ 1
## Model 2: y ~ x
##   Res.Df RSS Df Sum of Sq  F  Pr(>F)  
## 1      5  10                          
## 2      4   4  1         6  6 0.07048 .
## ---
## Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1




# compare a more complicated model with a simple model, using lrt
> lrtest(lm(y~1), lm(y~x))

## Likelihood ratio test
##
## Model 1: y ~ 1
## Model 2: y ~ x
##   #Df   LogLik Df  Chisq Pr(>Chisq)  
## 1   2 -10.0461                       
## 2   3  -7.2972  1 5.4977    0.01904 *
## ---
## Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


They give me different results. I was wondering why this is the case. Is it because of likelihood ratio test is based on MLE, which gives us a biased variance? Or is it because of asymptotic of lrt? More importantly, which one should I use?

I think I figured it out: The p-value calculated with the likelihood ratio test isn’t accurate because of the asymptotic property.
Here is a simulation:
# null model, generate data from the same distribution
Sampler <- function(n){

  data_random = rnorm(n)
  expgroup_random = rep(0:1, n/2)
  
  null_ll = logLik(lm(data_random ~ 1))
  full_ll = logLik(lm(data_random ~ expgroup_random))
  
  return(as.numeric(-2 * (null_ll - full_ll)))
} 


samples_size6 = c()
for (i in 1:10000){
    samples_size6[i] = Sampler(6)
}

# manually calculate chi from our data
chi = as.numeric(2 *(logLik(lm(y~x)) - logLik(lm(y~1))))

# calculate the frequency of more extreme events: random numbers greater than chi
sum(samples_size6 > chi)/10000

# 0.0728

Notice the p-value calculated by Monte Carlo is closer to the p-value from ANOVA (0.07048).
Besides, I also plotted the density when I set sample size = 4 and 10. Take a look:

Notice when sample size = 4, the density curve is quite different from the theoretical chi-square distribution (df = 1). Increasing the sample size would make the curve closer to the theoretical chi-square distribution.
Therefore, ANOVA is better when we have a small sample size.
