Recent work has explored the effect of statistics of language
model corpora on numerous downstream tasks. Findings
presented in Shin et al. (2022) demonstrate how the pre-
training corpus can impact few-shot performance, while
Razeghi et al. (2022) investigates how models are able to
perform numerical reasoning from in a few-shot setting. By
charting the performance of a arithmetic task given an input
operand and the frequency at which it is found in the pre-
training corpus, they concluded that accuracy tends to be
higher for terms that are found more frequently compared
to terms that are less frequent. Other works also suggest
that the pretraining corpus has a signiﬁcant impact on few-
shot behavior (Elazar et al., 2022; Kandpal et al., 2022).
These works observe a correlational and causal relationship
between the ability to answer factual questions and the fre-
quency of salient entities found in the pretraining corpus.
While the aforementioned works experiment with various
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling