First of all, we remark that the above generation is very different from the original ending of the story, which
provides a first evidence that it hasn’t simply been memorized (we discuss this much more thoroughly later on).
We also see that the generated content is consistent with the beginning in terms of plot, characters and events.
Arguably, this continuation is of higher quality than the one provided by GPT2-XL despite the size of the
model being almost 1000 times smaller.

However this raises the question whether the small models are effectively memorizing a small number of tem-
plates and the generation is very limited in terms of diversity, which would be a valid concern at this point. We
systematically address this later on, in Section 4.4. By testing the overlap of the generated stories with the train-
ing set and by considering out of distribution generation, we are led to the conclusion that those models do have
substantial diversity, and do not rely on memorization.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?