However, it is currently not clear whether the inability of SLMs to produce coherent text is a result of the
intrinsic complexity of natural language, or of the excessive breadth and diversity of the corpora used for training.
When we train a model on Wikipedia, for example, we are not only teaching it how to speak English, but also how
to encode and retrieve an immense amount of facts and concepts from various domains and disciplines. Could it
be that SLMs are overwhelmed by the amount and variety of information they have to process and store, and that
this hinders their ability to learn the core mechanisms and principles of language?
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?