2. The Pythia Suite
Following the advice of Birhane et al. (2021), in this section
we seek to explicitly document our choices, rationales, and
values in designing and implementing Pythia. As our goal is
to promote scientiﬁc research on large language models, we
prioritize consistency in model design and controlling for as
much potential sources of variation as possible rather than
trying to eek out the most performance from each model.
For example we use the parallel attention and feedforward
approach for all models, as it is becoming widely used for
the largest models, even though it is generally not recom-
mended for models with less than 2.7B parameters. To our
surprise, we ﬁnd that despite making choices we expect to

Pythia: A Suite for Analyzing Large Language Models
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling