Figure 6: Completion of different models - Example 1

Scores

Grammar: 6/10
Creativity: 3/10
Consistency: 2/10

Grammar: 5/10
Creativity: 6/10
Consistency: 3/10

Grammar: 7/10
Creativity: 5/10
Consistency: 5/10

Grammar: 9/10
Creativity: 6/10
Consistency: 9/10

Grammar: 8/10
Creativity: 3/10
Consistency: 7/10

Grammar: 7/10
Creativity: 6/10
Consistency: 4/10

4.3 Instruction-following examples and out-of-distribution generation

Table 12 provides an example of the generation of different models trained on the TinyStories-Instruct dataset,
together with the evaluation scores given by GPT-4. As the model size increases, we see an improvement both its
ability to follow instructions and to generate a coherent plot.

This dataset also enables us to test whether our models have a reasonable out of distribution performance.
Recall that in each entry of TinyStories-Instruct, the instructions are created as a (random) combination of possible

10

Model

Completion

Prompt

1M
8 layers
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?