But then, something unexpected happened. The car started to go faster and faster. The bus was scared. The car
said, ”Don’t worry, I will help you.” The car stopped and the bus was safe. The bus and the car became good
friends. They went fast on the road together and made everyone in the town happy.

Figure 13: Out-of-distribution generations: stories generated by a model trained on a dataset where the ”Words”
and ”Summary” parts do not appear together.

• Exact memorization: This is the simplest and most obvious form of memorization, where the model simply
copies an entire story or a large portion of it from the dataset, without changing anything. This can be easily
detected by checking the similarity or the hash of the generated story with the stories in the dataset.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?