statistics in the emergence of bias during pretraining.
We seek to investigate a counterfactual claim—if we were
to train our models on a corpus with different properties,
how would these models’ properties change downstream?
To test the effects of corpus statistics on the biases learned
by language models, we repeat segments of pretraining on
speciﬁc models, with altered corpus statistics. In particular,
for the Pythia-70M-deduped, Pythia-400M-deduped, Pythia-
1.4B-deduped, and Pythia-6.9B-deduped models, we take a
checkpoint and optimizer state 21B tokens (7%) prior to the
end of training, and resume training of the model such that
it sees the exact same data until the end of training, but with
morphologically masculine pronouns replaced by their fem-
inine counterparts. We also repeat this intervention for 63B
tokens (21%) prior to the end of training on just the Pythia-
1.4B-deduped model. We then measure model performance
on the WinoBias (Zhao et al., 2018) benchmark and the En-
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling