title: RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
summary: Layer-wise quantization is a key technique for efficiently compressing large
  models without expensive retraining. Previous methods typically quantize the
  weights of each layer by "uniformly" optimizing the layer reconstruction loss
  across all output tokens. However, in this paper, we demonstrate that
  better-quantized models can be obtained by prioritizing learning from important
  tokens (e.g. which have large attention scores). Building on this finding, we
  propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations
  (orthogonal transformation) to the model to mitigate outliers (those with
  exceptionally large magnitude), (2) scales the token feature based on its
  importance, and (3) quantizes the model using the GPTQ framework with the
  second-order statistics computed by scaled tokens. To compute token importance,
  we explore both heuristic and dynamic strategies. Based on a thorough analysis
  of all approaches, we adopt attention concentration, which uses attention
  scores of each token as its importance, as the best approach. We demonstrate
  that RSQ consistently outperforms baseline methods across multiple downstream
  tasks and three model families: LLaMA3, Mistral, and Qwen2.5. Additionally,
  models quantized with RSQ achieve superior performance on long-context tasks,
  further highlighting its effectiveness. Lastly, RSQ demonstrates
  generalizability across various setups, including different model sizes,
  calibration datasets, bit precisions, and quantization methods.
paper_authors._id: 67c7575d79b553252b95b0d6
paper_authors.name: Yi-Lin Sung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T21:15:30.002Z
paper_authors._id: 654ffe334d9e71e17becc660
paper_authors.avatarUrl: /avatars/022b7a77051d26c4e5cbf254b7352eb9.svg
paper_authors.fullname: Yi-Lin Sung
paper_authors.type: user
paper_authors.user: a2889184
paper_authors._id: 67c7575d79b553252b95b0d7
paper_authors.name: Prateek Yadav
paper_authors._id: 67c7575d79b553252b95b0d8
paper_authors.name: Jialu Li
paper_authors._id: 67c7575d79b553252b95b0d9
paper_authors.name: Jaehong Yoon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T21:15:27.436Z
paper_authors._id: 652066649004117947e46ed6
paper_authors.avatarUrl: /avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
paper_authors.fullname: Jaehong Yoon
paper_authors.type: user
paper_authors.user: jaehong31
paper_authors._id: 67c7575d79b553252b95b0da
paper_authors.name: Mohit Bansal
paper_title: RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
paper_summary: Layer-wise quantization is a key technique for efficiently compressing large
  models without expensive retraining. Previous methods typically quantize the
  weights of each layer by "uniformly" optimizing the layer reconstruction loss
  across all output tokens. However, in this paper, we demonstrate that
  better-quantized models can be obtained by prioritizing learning from important
  tokens (e.g. which have large attention scores). Building on this finding, we
  propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations
  (orthogonal transformation) to the model to mitigate outliers (those with
  exceptionally large magnitude), (2) scales the token feature based on its
  importance, and (3) quantizes the model using the GPTQ framework with the
  second-order statistics computed by scaled tokens. To compute token importance,
  we explore both heuristic and dynamic strategies. Based on a thorough analysis
  of all approaches, we adopt attention concentration, which uses attention
  scores of each token as its importance, as the best approach. We demonstrate
  that RSQ consistently outperforms baseline methods across multiple downstream
  tasks and three model families: LLaMA3, Mistral, and Qwen2.5. Additionally,
  models quantized with RSQ achieve superior performance on long-context tasks,
  further highlighting its effectiveness. Lastly, RSQ demonstrates
  generalizability across various setups, including different model sizes,
  calibration datasets, bit precisions, and quantization methods.
paper_ai_keywords: layer-wise quantization layer reconstruction loss attention scores orthogonal transformation token feature scaling GPTQ framework second-order statistics attention concentration LLaMA3 Mistral Qwen2.5 long-context tasks