Softmax Attention with Constant Cost per Token
<|reference_start|>Softmax Attention with Constant Cost per Token: We propose a simple modification to the conventional attention mechanism applied by Transformers: Instead of quantifying pairwise query-key similarity with scaled dot-products, we quantify it with the logarithms of scaled dot-products of exponentials. Our modification linearizes attention with exponential kernel feature maps, whose corresponding feature function is infinite dimensional. We show that our modification is expressible as a composition of log-sums of exponentials, with a latent space of constant size, enabling application with constant time and space complexity per token. We implement our modification, verify that it works in practice, and conclude that it is a promising alternative to conventional attention.<|reference_end|>
@article{heinsen2024softmax,
    title={Softmax Attention with Constant Cost per Token},
    author={Franz A. Heinsen},
    journal={arXiv preprint arXiv:2404.05843},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.05843},
    primaryClass={cs.LG cs.CL}
}