Memorization is a Poisson Point Process Building on
the extensive literature on memorization in large language
models (Carlini et al., 2019; 2021; Hu et al., 2022), we ask
the following question: does the location of a particular
sequence in the training dataset inﬂuence the likelihood
of it being memorized? Leveraging Pythia’s reproducible

dataloader setup we answer this question in the negative,
and furthermore ﬁnd that a poisson point process is a very
good model for the occurrence of memorized sequences
over the course of training.
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling