A phase transition between positional and semantic learning in a solvable model of dot-product attention
<|reference_start|>A phase transition between positional and semantic learning in a solvable model of dot-product attention: We investigate how a dot-product attention layer learns a positional attention matrix (with tokens attending to each other based on their respective positions) and a semantic attention matrix (with tokens attending to each other based on their meaning). For an algorithmic task, we experimentally show how the same simple architecture can learn to implement a solution using either the positional or semantic mechanism. On the theoretical side, we study the learning of a non-linear self-attention layer with trainable tied and low-rank query and key matrices. In the asymptotic limit of high-dimensional data and a comparably large number of training samples, we provide a closed-form characterization of the global minimum of the non-convex empirical loss landscape. We show that this minimum corresponds to either a positional or a semantic mechanism and evidence an emergent phase transition from the former to the latter with increasing sample complexity. Finally, we compare the dot-product attention layer to linear positional baseline, and show that it outperforms the latter using the semantic mechanism provided it has access to sufficient data.<|reference_end|>
@article{cui2024a,
    title={A phase transition between positional and semantic learning in a
  solvable model of dot-product attention},
    author={Hugo Cui, Freya Behrens, Florent Krzakala, Lenka Zdeborov'a},
    journal={arXiv preprint arXiv:2402.03902},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03902},
    primaryClass={cs.LG}
}