Self-attention as an attractor network: transient memories without backpropagation
<|reference_start|>Self-attention as an attractor network: transient memories without backpropagation: Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics community as it can be written as the derivative of an energy function in certain cases: while it is possible to write the cross-attention layer as a modern Hopfield network, the same is not possible for the self-attention, which is used in the GPT architectures and other autoregressive models. In this work we show that it is possible to obtain the self-attention layer as the derivative of local energy terms, which resemble a pseudo-likelihood. We leverage the analogy with pseudo-likelihood to design a recurrent model that can be trained without backpropagation: the dynamics shows transient states that are strongly correlated with both train and test examples. Overall we present a novel framework to interpret self-attention as an attractor network, potentially paving the way for new theoretical approaches inspired from physics to understand transformers.<|reference_end|>
@article{d'amico2024self-attention,
    title={Self-attention as an attractor network: transient memories without
  backpropagation},
    author={Francesco D'Amico, Matteo Negri},
    journal={arXiv preprint arXiv:2409.16112},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.16112},
    primaryClass={cs.LG cond-mat.dis-nn}
}