title: Matryoshka Quantization
summary: Quantizing model weights is critical for reducing the communication and
  inference costs of large models. However, quantizing models -- especially to
  low precisions like int4 or int2 -- requires a trade-off in model quality;
  int2, in particular, is known to severely degrade model quality. Consequently,
  practitioners are often forced to maintain multiple models with different
  quantization levels or serve a single model that best satisfies the
  quality-latency trade-off. On the other hand, integer data types, such as int8,
  inherently possess a nested (Matryoshka) structure where smaller bit-width
  integers, like int4 or int2, are nested within the most significant bits. This
  paper proposes Matryoshka Quantization (MatQuant), a novel multi-scale
  quantization technique that addresses the challenge of needing multiple
  quantized models. It allows training and maintaining just one model, which can
  then be served at different precision levels. Furthermore, due to the
  co-training and co-distillation regularization provided by MatQuant, the int2
  precision models extracted by MatQuant can be up to 10% more accurate than
  standard int2 quantization (using techniques like QAT or OmniQuant). This
  represents significant progress in model quantization, demonstrated by the fact
  that, with the same recipe, an int2 FFN-quantized Gemma-2 9B model is more
  accurate than an int8 FFN-quantized Gemma-2 2B model.
paper_authors._id: 67aae91b83b1182df7c0cf54
paper_authors.name: Pranav Nair
paper_authors._id: 67aae91b83b1182df7c0cf55
paper_authors.name: Puranjay Datta
paper_authors._id: 67aae91b83b1182df7c0cf56
paper_authors.name: Jeff Dean
paper_authors._id: 67aae91b83b1182df7c0cf57
paper_authors.name: Prateek Jain
paper_authors._id: 67aae91b83b1182df7c0cf58
paper_authors.name: Aditya Kusupati
paper_title: Matryoshka Quantization
paper_summary: Quantizing model weights is critical for reducing the communication and
  inference costs of large models. However, quantizing models -- especially to
  low precisions like int4 or int2 -- requires a trade-off in model quality;
  int2, in particular, is known to severely degrade model quality. Consequently,
  practitioners are often forced to maintain multiple models with different
  quantization levels or serve a single model that best satisfies the
  quality-latency trade-off. On the other hand, integer data types, such as int8,
  inherently possess a nested (Matryoshka) structure where smaller bit-width
  integers, like int4 or int2, are nested within the most significant bits. This
  paper proposes Matryoshka Quantization (MatQuant), a novel multi-scale
  quantization technique that addresses the challenge of needing multiple
  quantized models. It allows training and maintaining just one model, which can
  then be served at different precision levels. Furthermore, due to the
  co-training and co-distillation regularization provided by MatQuant, the int2
  precision models extracted by MatQuant can be up to 10% more accurate than
  standard int2 quantization (using techniques like QAT or OmniQuant). This
  represents significant progress in model quantization, demonstrated by the fact
  that, with the same recipe, an int2 FFN-quantized Gemma-2 9B model is more
  accurate than an int8 FFN-quantized Gemma-2 2B model.
paper_ai_keywords: quantizing model weights Matryoshka Quantization (MatQuant) multi-scale quantization co-training co-distillation regularization FFN-quantized