title: Quamba2: A Robust and Scalable Post-training Quantization Framework for
    Selective State Space Models
summary: State Space Models (SSMs) are emerging as a compelling alternative to
  Transformers because of their consistent memory usage and high performance.
  Despite this, scaling up SSMs on cloud services or limited-resource devices is
  challenging due to their storage requirements and computational power. To
  overcome this, quantizing SSMs with low bit-width data formats can reduce model
  size and benefit from hardware acceleration. As SSMs are prone to
  quantization-induced errors, recent efforts have focused on optimizing a
  particular model or bit-width for efficiency without sacrificing performance.
  However, distinct bit-width configurations are essential for different
  scenarios, like W4A8 for boosting large-batch decoding speed, and W4A16 for
  enhancing generation speed in short prompt applications for a single user. To
  this end, we present Quamba2, compatible with W8A8, W4A8, and W4A16 for both
  Mamba1 and Mamba2 backbones, addressing the growing demand for SSM deployment
  on various platforms. Based on the channel order preserving and activation
  persistence of SSMs, we propose an offline approach to quantize inputs of a
  linear recurrence in 8-bit by sorting and clustering for input x, combined
  with a per-state-group quantization for input-dependent parameters B and C.
  To ensure compute-invariance in the SSM output, we rearrange weights offline
  according to the clustering sequence. The experiments show that Quamba2-8B
  outperforms several state-of-the-art SSM quantization methods and delivers
  1.3times and 3times speed-ups in the pre-filling and generation stages,
  respectively, while offering 4times memory reduction with only a 1.6%
  average accuracy drop. The evaluation on MMLU shows the generalizability and
  robustness of our framework. The code and quantized models will be released at:
  https://github.com/enyac-group/Quamba.
paper_authors._id: 67eea40fe01fcf131baa149e
paper_authors.name: Hung-Yueh Chiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T19:20:45.914Z
paper_authors._id: 65ca7d8d5cf913133d3b493a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ca7d8d5cf913133d3b493a/i9qx6xdYeGSuwsxgQCb7N.png
paper_authors.fullname: Hung-Yueh Chiang
paper_authors.type: user
paper_authors.user: hychiang
paper_authors._id: 67eea40fe01fcf131baa149f
paper_authors.name: Chi-Chih Chang
paper_authors._id: 67eea40fe01fcf131baa14a0
paper_authors.name: Natalia Frumkin
paper_authors._id: 67eea40fe01fcf131baa14a1
paper_authors.name: Kai-Chiang Wu
paper_authors._id: 67eea40fe01fcf131baa14a2
paper_authors.name: Mohamed S. Abdelfattah
paper_authors._id: 67eea40fe01fcf131baa14a3
paper_authors.name: Diana Marculescu
paper_title: Quamba2: A Robust and Scalable Post-training Quantization Framework for
    Selective State Space Models
paper_summary: State Space Models (SSMs) are emerging as a compelling alternative to
  Transformers because of their consistent memory usage and high performance.
  Despite this, scaling up SSMs on cloud services or limited-resource devices is
  challenging due to their storage requirements and computational power. To
  overcome this, quantizing SSMs with low bit-width data formats can reduce model
  size and benefit from hardware acceleration. As SSMs are prone to
  quantization-induced errors, recent efforts have focused on optimizing a
  particular model or bit-width for efficiency without sacrificing performance.
  However, distinct bit-width configurations are essential for different
  scenarios, like W4A8 for boosting large-batch decoding speed, and W4A16 for
  enhancing generation speed in short prompt applications for a single user. To
  this end, we present Quamba2, compatible with W8A8, W4A8, and W4A16 for both
  Mamba1 and Mamba2 backbones, addressing the growing demand for SSM deployment
  on various platforms. Based on the channel order preserving and activation
  persistence of SSMs, we propose an offline approach to quantize inputs of a
  linear recurrence in 8-bit by sorting and clustering for input x, combined
  with a per-state-group quantization for input-dependent parameters B and C.
  To ensure compute-invariance in the SSM output, we rearrange weights offline
  according to the clustering sequence. The experiments show that Quamba2-8B
  outperforms several state-of-the-art SSM quantization methods and delivers
  1.3times and 3times speed-ups in the pre-filling and generation stages,
  respectively, while offering 4times memory reduction with only a 1.6%
  average accuracy drop. The evaluation on MMLU shows the generalizability and
  robustness of our framework. The code and quantized models will be released at:
  https://github.com/enyac-group/Quamba.
paper_ai_summary: Quamba2 quantizes state space models (SSMs) with multi-bit-width configurations to enhance deployment on diverse platforms, offering speed-ups and memory reduction with minimal performance loss.
paper_ai_keywords: state space models (SSMs) quantization low bit-width data formats hardware acceleration quantization-induced errors W4A8 W4A16 Mamba1 Mamba2 channel order preserving activation persistence offline approach linear recurrence per-state-group quantization compute-invariance speed-ups memory reduction MMLU Quamba2-8B