Climbing the Complexity Ladder with Expressive Attention
<|reference_start|>Climbing the Complexity Ladder with Expressive Attention: Attention involves comparing query and key vectors in terms of a scalar product, $\mathbf{Q}^T\mathbf{K}$, together with a subsequent softmax normalization. Classicaly, parallel/orthogonal/antiparallel queries and keys lead to large/intermediate/small attention weights. Here we study expressive attention (EA), which is based on $(\mathbf{Q}^T\mathbf{K})^2$, the squared dot product. In this case attention is enhanced when query and key are either parallel or antiparallel, and suppressed for orthogonal configurations. For a series of autoregressive prediction tasks, we find that EA performs at least as well as the standard mechanism, dot-product attention (DPA). Increasing task complexity, EA is observed to outperform DPA with increasing margins, which also holds for multi-task settings. For a given model size, EA manages to achieve 100\% performance for a range of complexity levels not accessible to DPA.<|reference_end|>
@article{gros2024climbing,
    title={Climbing the Complexity Ladder with Expressive Attention},
    author={Claudius Gros},
    journal={arXiv preprint arXiv:2407.18601},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.18601},
    primaryClass={cs.LG cs.AI}
}