To this end, we rely on the latest text generation models by OpenAI (GPT-3.5 and GPT-4) which are able
to produce large amounts of synthetic content according to instructions. In particular, we instruct the models to
produce content that only uses vocabulary that a typical 3-year-old child would understand. We restrict the content
to have the format of short stories in English. The main challenge in using large language models for producing
training data is generating a dataset that is sufficiently diverse: prompting those models to produce stories, even
if the temperature of generation is set to a high value, will still produce a very repetitive dataset, whose diversity
is very far from what is required for training a language model that has a comparable ”understanding” of language
to that of children.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?