3-level meta-analysis: Level 3 (study level) has a variance of 0, should I drop this level? I am doing a 3-level meta-analysis.
First, I fit a theoretically-driven model, where level 1 is the sampling variance, level 2 is the effect level, level 3 is the study level. This method is well known (e.g. see Cheung, 2019). I found that the level 3 variance is basically 0. Hence, I dropped this level and fit a 2-level model (level 1 is the sampling variance, level 2 is the effect level). Comparing model fit, the 2-level model has a slightly better fit (based on AIC and BIC), but only very slightly (e.g. AIC: -8 vs -12)
Question:
Should I go with the 2-level or 3-level model? Another way of asking is: should the model structure be based on theory or model fit?
Thanks in advance!
Ref:
Cheung, M.WL. A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes. Neuropsychol Rev 29, 387–396 (2019).  10.1007/s11065-019-09415-6

Theory, particularly for a meta-analysis and where the fit is comparable as it is here. If the additional level is reasonable eg. where there is a clear nesting of studies according to the random effect and it seems reasonable to account for this, then I would stick to the three-level error structure regardless of the fit statistics.
If the findings differ substantially between the two, you can always report the results of both models.
