• Diversity of instructions: Recall that in the TinyStories-Instruct dataset, we provide a set of instructions in the
form of summaries or words contained in the stories, followed by the stories themselves. We can then change
the instructions, verify that the combinations do not appear in the dataset and see how the models adapt to
the new instructions. We find that the models can generate diverse stories that follow the instructions, even
if they are novel or challenging, such as requiring the model to fit unlikely words into the story or adding
features such as a plot twist or a bad ending.

4.4.1 Quantitative measurement of similarity using Rouge score.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?