24

and layers.

8 Conclusion

In this work, we have presented TinyStories, a synthetic dataset of short stories that only contain words that a typical
3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We have shown that TinyStories can be
used to train and evaluate small language models (SLMs) that are much smaller than the state-of-the-art models,
yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect
grammar, and demonstrate reasoning capabilities.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?