title: Model-Preserving Adaptive Rounding
summary: The main goal of post-training quantization (PTQ) is to produced a compressed
  model whose output distribution is as close to the original model's as
  possible. To do this tractably, almost all LLM PTQ algorithms quantize linear
  layers by independently minimizing the immediate activation error. However,
  this localized objective ignores the effect of subsequent layers, so reducing
  it does not necessarily give a closer model. In this work, we introduce Yet
  Another Quantization Algorithm (YAQA), an adaptive rounding algorithm that uses
  Kronecker-factored approximations of each linear layer's Hessian with respect
  to the full model KL divergence. YAQA consists of two components:
  Kronecker-factored sketches of the full layerwise Hessian that can be tractably
  computed for hundred-billion parameter LLMs, and a quantizer-independent
  rounding algorithm that uses these sketches and comes with theoretical
  guarantees. Across a wide range of models and quantizers, YAQA empirically
  reduces the KL divergence to the original model by approx 30% while
  achieving state of the art performance on downstream tasks.
paper_authors._id: 683a57be57009ee6f3c7a0d2
paper_authors.name: Albert Tseng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:43:31.763Z
paper_authors._id: 649e3f263914db6cf8e8ab1f
paper_authors.avatarUrl: /avatars/35298884b7633c67a9bcff0a32d31211.svg
paper_authors.fullname: Albert Tseng
paper_authors.type: user
paper_authors.user: at676
paper_authors._id: 683a57be57009ee6f3c7a0d3
paper_authors.name: Zhaofeng Sun
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-31T01:13:35.081Z
paper_authors._id: 67aee52fe680b6d64f1d1646
paper_authors.avatarUrl: /avatars/334b73779ce2e62c41071d91cc1ffc5e.svg
paper_authors.fullname: Zhaofeng Sun
paper_authors.type: user
paper_authors.user: thuEcstasy1
paper_authors._id: 683a57be57009ee6f3c7a0d4
paper_authors.name: Christopher De Sa
paper_title: Model-Preserving Adaptive Rounding
paper_summary: The main goal of post-training quantization (PTQ) is to produced a compressed
  model whose output distribution is as close to the original model's as
  possible. To do this tractably, almost all LLM PTQ algorithms quantize linear
  layers by independently minimizing the immediate activation error. However,
  this localized objective ignores the effect of subsequent layers, so reducing
  it does not necessarily give a closer model. In this work, we introduce Yet
  Another Quantization Algorithm (YAQA), an adaptive rounding algorithm that uses
  Kronecker-factored approximations of each linear layer's Hessian with respect
  to the full model KL divergence. YAQA consists of two components:
  Kronecker-factored sketches of the full layerwise Hessian that can be tractably
  computed for hundred-billion parameter LLMs, and a quantizer-independent
  rounding algorithm that uses these sketches and comes with theoretical
  guarantees. Across a wide range of models and quantizers, YAQA empirically
  reduces the KL divergence to the original model by approx 30% while
  achieving state of the art performance on downstream tasks.
paper_ai_summary: YAQA, an adaptive rounding algorithm using Kronecker-factored approximations of the full model's Hessian, reduces KL divergence and improves performance for post-training quantization of large language models.
paper_ai_keywords: post-training quantization LLM PTQ linear layers activation error Kronecker-factored approximations Hessian full model KL divergence quantizer-independent rounding algorithm theoretical guarantees