We claim that our models are not doing exact memorization or simple template matching, as evidenced by the
methods and metrics we use to evaluate the diversity of the content generated by the models. We rely on several
approaches:

• Manual inspection: We generate completions for a range of human-constructed stories. We inspect the stories
generated by the models and check that they are not copies or close modifications of the stories in the dataset.
• Completion of training stories: We take stories from the training set, truncate them in the middle and generate
alternative completions with our models. We then compare the completions with the original stories. We
observe that the completions are typically very different from the original stories, and often introduce new
characters, events, or twists. This is shown in Figure 14.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?