We train models of various sizes and architectures on TinyStories. For each amount of flops, we select the model
and the number of training steps that achieve the lowest validation loss among the possible combinations. We vary
the number of layers from 2, 4, 8, 12 and the hidden dimension from 64, 128, 256, 512, 768, 1024, 2048. The result is
shown in Figure 6. Although the number of points may be a bit small for the data to be very conclusive, the plot
points to a polynomial dependence.

Choosing the number of heads. Another design choice for transformers is the number of attention heads for
each layer. It is not obvious how the number of heads affects the performance of the model, given a fixed model
width and depth. Our results, shown in Figure 24, suggest that in the regime where the number of heads is small,
increasing it improves the performance of the model across all metrics.

7 Related Works
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?