Attention layers provably solve single-location regression
Pierre Marion Raphaël Berthier Gérard Biau Claire Boyer
Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.
theory of neural networks, attention, Transformer, statistical learning theory, optimization, first-oder optimization, gradient flow
We introduce single-location regression, a novel regression task, and show that attention solves this task.