FAST: Factorizable Attention for Speeding up Transformers
<|reference_start|>FAST: Factorizable Attention for Speeding up Transformers: Motivated by the factorization inherent in the original fast multipole method and the improved fast Gauss transform we introduce a factorable form of attention that operates efficiently in high dimensions. This approach reduces the computational and memory complexity of the attention mechanism in transformers from $O(N^2)$ to $O(N)$. In comparison to previous attempts, our work presents a linearly scaled attention mechanism that maintains the full representation of the attention matrix without compromising on sparsification and incorporates the all-to-all relationship between tokens. We explore the properties of our new attention metric and conduct tests in various standard settings. Results indicate that our attention mechanism has a robust performance and holds significant promise for diverse applications where self-attention is used.<|reference_end|>
@article{gerami2024fast:,
    title={FAST: Factorizable Attention for Speeding up Transformers},
    author={Armin Gerami, Monte Hoover, Pranav S. Dulepet, Ramani Duraiswami},
    journal={arXiv preprint arXiv:2402.07901},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07901},
    primaryClass={cs.LG cs.AI cs.NA math.NA}
}