Clustering in pure-attention hardmax transformers and its role in sentiment analysis
<|reference_start|>Clustering in pure-attention hardmax transformers and its role in sentiment analysis: Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.<|reference_end|>
@article{alcalde2024clustering,
    title={Clustering in pure-attention hardmax transformers and its role in
  sentiment analysis},
    author={Albert Alcalde, Giovanni Fantuzzi, Enrique Zuazua},
    journal={arXiv preprint arXiv:2407.01602},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01602},
    primaryClass={cs.CL cs.LG math.DS stat.ML}
}