Attention is Naturally Sparse with Gaussian Distributed Input
<|reference_start|>Attention is Naturally Sparse with Gaussian Distributed Input: The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures. Addressing this, sparse attention emerges as a key innovation, aiming to reduce computational load while maintaining model performance. This study presents a rigorous theoretical analysis of the sparsity in attention scores within LLMs, particularly under the framework of Gaussian inputs. By establishing a set of foundational assumptions and employing a methodical theoretical approach, we unravel the intrinsic characteristics of attention score sparsity and its implications on computational efficiency. Our main contribution lies in providing a detailed theoretical examination of how sparsity manifests in attention mechanisms, offering insights into the potential trade-offs between computational savings and model effectiveness. This work not only advances our understanding of sparse attention but also provides a scaffold for future research in optimizing the computational frameworks of LLMs, paving the way for more scalable and efficient AI systems.<|reference_end|>
@article{deng2024attention,
    title={Attention is Naturally Sparse with Gaussian Distributed Input},
    author={Yichuan Deng, Zhao Song, Chiwun Yang},
    journal={arXiv preprint arXiv:2404.02690},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.02690},
    primaryClass={cs.LG cs.AI cs.CL}
}