2.5. Evaluation

While the primary focus of this work is to promote scien-
tiﬁc research on the behaviors of large language models,
and state-of-the-art performance is not necessarily a core
requirement, we ﬁnd that Pythia and Pythia (Deduplicated)
perform very similarly to OPT and BLOOM models on a
variety of NLP benchmarks. These results are presented in
Appendix G. We use the Language Model Evaluation Har-
ness (Gao et al., 2021) to run evaluations on eight common
language modeling benchmarks: OpenAI’s LAMBADA
variant, PIQA, the Winograd Schema Challenge, Wino-
Grande, ARC (easy and challenge sets separately), SciQ,
and LogiQA. We consistently ﬁnd that Pythia and Pythia
(Deduplicated) perform very similarly to OPT and BLOOM
models.

2.6. Novel Observations in Evaluation
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling