Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
<|reference_start|>Mamba-PTQ: Outlier Channels in Recurrent Large Language Models: Modern recurrent layers are emerging as a promising path toward edge deployment of foundation models, especially in the context of large language models (LLMs). Compressing the whole input sequence in a finite-dimensional representation enables recurrent layers to model long-range dependencies while maintaining a constant inference cost for each token and a fixed memory requirement. However, the practical deployment of LLMs in resource-limited environments often requires further model compression, such as quantization and pruning. While these techniques are well-established for attention-based models, their effects on recurrent layers remain underexplored.   In this preliminary work, we focus on post-training quantization for recurrent LLMs and show that Mamba models exhibit the same pattern of outlier channels observed in attention-based LLMs. We show that the reason for the difficulty of quantizing SSMs is caused by activation outliers, similar to those observed in transformer-based LLMs. We report baseline results for post-training quantization of Mamba that do not take into account the activation outliers and suggest first steps for outlier-aware quantization.<|reference_end|>
@article{pierro2024mamba-ptq:,
    title={Mamba-PTQ: Outlier Channels in Recurrent Large Language Models},
    author={Alessandro Pierro, Steven Abreu},
    journal={arXiv preprint arXiv:2407.12397},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12397},
    primaryClass={cs.LG cs.AI cs.NE}
}