Figure 19: Multi-scale distance-based attention.

Semantic based attention. We also observe that there is (1). one head that the word “the” and “a” all attend
to the word “banana”, interestingly, the “the” at “the park” also attends to “banana”, but the model still manage
to generate “park”, which is the consistent completion. (2). Another attention head gives a pattern where the
tokens “the” and “a” all attend to “park”. (3). There is third head that most of the words attend to the name of
“Tom” and “Lucy”.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?