Completion by a 28M-
parameters model trained
on TinyStories

very bitter. She does not like it. She says, “I don’t like this soup. It is too bitter.”
Tom is sorry. He says, “I’m sorry, Jane. I didn’t know you don’t like bitter soup. I will make you something
else.” He looks around the kitchen and finds some bread and cheese. He puts them on the table and says,
“Here, Jane. You can have some bread and cheese. They are not bitter. They are sweet and yummy.”
Jane is happy. She says, “Thank you, Tom. You are a good friend. I like bread and cheese. They are not
bitter.”

Figure 1: Comparison between the completions given by a 28M-parameters model trained on the TinyStories dataset
and GPT-2XL (1.5B)
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?