dataset. Performance is averaged over a group of log-spaced
bins.
We observe that for both arithmetic and QA experiments,
model sizes affect the correlation between average perfor-
mance and the term frequencies, indicating that this corre-
lation is an emergent property in larger models. Smaller
models rarely produce accurate results on the task despite
being given up to 16 few-shot examples, as shown in Fig-
ure 4, where models at sizes below 1 billion are unable to
perform well even in later stages of training, suggesting
that these models are not successful at learning the required
pattern-matching to solve the tasks regardless of frequency
of pertinent information in their training data. Similar pat-
terns can be seen in Figure 5 where performance increase as
training progresses mainly happens for larger models only.
For the multiplication arithmetic task, we also calculate the
performance discrepancy between the top 10% most fre-
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling