data. As the evaluation was performed on the ﬁrst 64 tokens
of every sequence within the training corpus, in the same
order of training, we can consider each batch to represent a
hypothetical time interval, where a unit of time corresponds
to a sequence of the training corpus.
Our sample distribution, in this case, is the observed distri-
bution of sample points deﬁned as the number of memorized
sequences in a batch of training data. The theoretical dis-
tribution is the best ﬁt Poisson distribution from samples.
We use a batch size of 512 sequences for these plots, we
observe similar results for various batch sizes
The count (color bar to the right in Figure 3) indicates the
density of plotted points (also indicated by size) on the Q-Q
plot. Q-Q plots serve the purpose of being a goodness of
ﬁt test for asserting the fact that the rate of occurrence of
memorized sequences in training data is uniform.
This ﬁnding is important for practitioners seeking to control
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling