Pythia: A Suite for Analyzing Large Language Models

other suite of publicly available language models.
We train all models for 299,892,736,000 ≈ 300B tokens.
This equates to 1 epoch on the original Pile, and ≈1.5
epochs on the deduplicated Pile, which is 207B tokens in
size. We refer to the models trained on the original Pile
as “Pythia-xxx”, where ‘xxx’ is the model’s total parameter
count rounded to 2 signiﬁcant ﬁgures, and their counterparts
trained on the deduplicated Pile as “Pythia-xxx-deduped”.

2.5. Evaluation
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling