tion, the average accuracy of all prompts with the same x1
over all values of x2 is mapped to the number of times x1
is found in the sampled pretraining data that each evaluated
model checkpoint sees. In few-shot settings, we sample
examples with digits that differ from the x1 values being
measured.
As a QA task, we use TriviaQA (Joshi et al., 2017) with
a simple template of “Q: x1 \n A: y” with x1 being the
question and y answer, where y is included for a few-shot
sample or left blank for the sample being evaluated. The
model prediction is evaluated with exact match over the
set of possible answers. The term frequencies of a single
question-answer pair (“QA pair”) are calculated based on
the number of times all salient entities of that QA pair appear
in a sampled pretraining data sequence seen by a given
checkpoint. We follow the original experiment using 4 shots
and evaluate both the training and the validation split of the
dataset. Performance is averaged over a group of log-spaced
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling