One day, Lucy asks Tom: ”I am looking for a banana but I can’t find it”. Tom says: ”Don’t worry, I will help you”.
Lucy and Tom go to the park. They look for the banana together. After a while, they found the banana. Lucy is happy. She says:
”Thank you, Tom. You are a good friend.” Tom: ”You are welcome, Lucy. I am happy to help you. Let’s eat the banana together!”

20

There seems to be a clear separation between heads with attention pattern based mainly on the distance between

tokens, and heads whose attention pattern has a stronger dependence on the semantic meaning:

Distance based attention. Out of the 16 attention heads, we observe multiple positional-based attention heads,
such that each token attends to tokens with a prescribed relative distance. Different heads are associated with
different distances.

Figure 19: Multi-scale distance-based attention.
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?