Figure 22: Tokens which induce high activations to different neurons in GPT-XL

performance. TinyStories is such a dataset, as it enables us to train and evaluate LMs that are orders of magnitude
smaller than the state-of-the-art models, yet still have the basic capability of producing coherent text.

In this work, we take the first steps towards using TinyStories as a testbed for exploring architectures and
hyperparameters for NLP. We show that our small models exhibit some similar patterns to the ones observed in
LLMs in certain aspects. In particular, we investigate two questions: how to balance model size and learning budget
for a fixed amount of training flops, and how to choose the number of attention heads for a given model width and
depth.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?