6 Exploring architectures and hyperparameters for NLP with TinyS-

tories

One of the main challenges in developing large language models (LLMs) comes from the high computational cost
involved in training. Finding the best architectures, training algorithms and hyperparameters for LLMs requires a
lot of resources and experimentation. Therefore, it would be useful to have a smaller and simpler dataset that can
still capture some of the basic capabilities of LLMs, and allow us to study how different design choices affect their

10The rationale behind choosing the penultimate layer is that tokens have already been processed by most layers at this point. We
take the before-last rather than the last layer since the hidden representation in the last layer only has only the role of predicting the
next token, so information may be lost at that point.

22

Layer #13, Neuron #1

Layer #12, Neuron #2

It’s not safe to play in the fog.

She liked to keep her toys and books in the right place.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?