The Hidden Attention of Mamba Models
<|reference_start|>The Hidden Attention of Mamba Models: The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and computer vision. Selective SSMs are viewed as dual models, in which one trains in parallel on the entire sequence via an IO-aware parallel scan, and deploys in an autoregressive manner. We add a third view and show that such models can be viewed as attention-driven models. This new perspective enables us to empirically and theoretically compare the underlying mechanisms to that of the self-attention layers in transformers and allows us to peer inside the inner workings of the Mamba model with explainability methods. Our code is publicly available.<|reference_end|>
@article{ali2024the,
    title={The Hidden Attention of Mamba Models},
    author={Ameen Ali, Itamar Zimerman, Lior Wolf},
    journal={arXiv preprint arXiv:2403.01590},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.01590},
    primaryClass={cs.LG}
}