title: Unified Scaling Laws for Compressed Representations
summary: Scaling laws have shaped recent advances in machine learning by enabling
  predictable scaling of model performance based on model size, computation, and
  data volume. Concurrently, the rise in computational cost for AI has motivated
  model compression techniques, notably quantization and sparsification, which
  have emerged to mitigate the steep computational demands associated with
  large-scale training and inference. This paper investigates the interplay
  between scaling laws and compression formats, exploring whether a unified
  scaling framework can accurately predict model performance when training occurs
  over various compressed representations, such as sparse, scalar-quantized,
  sparse-quantized or even vector-quantized formats. Our key contributions
  include validating a general scaling law formulation and showing that it is
  applicable both individually but also composably across compression types.
  Based on this, our main finding is demonstrating both theoretically and
  empirically that there exists a simple "capacity" metric -- based on the
  representation's ability to fit random Gaussian data -- which can robustly
  predict parameter efficiency across multiple compressed representations. On the
  practical side, we extend our formulation to directly compare the accuracy
  potential of different compressed formats, and to derive better algorithms for
  training over sparse-quantized formats.
paper_authors._id: 683ee1a6e65df2618201e81b
paper_authors.name: Andrei Panferov
paper_authors._id: 683ee1a6e65df2618201e81c
paper_authors.name: Alexandra Volkova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:09.324Z
paper_authors._id: 65f08bd773f2e1e291244d86
paper_authors.avatarUrl: /avatars/4d66171163094024ebc82bb59859d9c5.svg
paper_authors.fullname: Alexandra
paper_authors.type: user
paper_authors.user: alexandraww
paper_authors._id: 683ee1a6e65df2618201e81d
paper_authors.name: Ionut-Vlad Modoranu
paper_authors._id: 683ee1a6e65df2618201e81e
paper_authors.name: Vage Egiazarian
paper_authors._id: 683ee1a6e65df2618201e81f
paper_authors.name: Mher Safaryan
paper_authors._id: 683ee1a6e65df2618201e820
paper_authors.name: Dan Alistarh
paper_title: Unified Scaling Laws for Compressed Representations
paper_summary: Scaling laws have shaped recent advances in machine learning by enabling
  predictable scaling of model performance based on model size, computation, and
  data volume. Concurrently, the rise in computational cost for AI has motivated
  model compression techniques, notably quantization and sparsification, which
  have emerged to mitigate the steep computational demands associated with
  large-scale training and inference. This paper investigates the interplay
  between scaling laws and compression formats, exploring whether a unified
  scaling framework can accurately predict model performance when training occurs
  over various compressed representations, such as sparse, scalar-quantized,
  sparse-quantized or even vector-quantized formats. Our key contributions
  include validating a general scaling law formulation and showing that it is
  applicable both individually but also composably across compression types.
  Based on this, our main finding is demonstrating both theoretically and
  empirically that there exists a simple "capacity" metric -- based on the
  representation's ability to fit random Gaussian data -- which can robustly
  predict parameter efficiency across multiple compressed representations. On the
  practical side, we extend our formulation to directly compare the accuracy
  potential of different compressed formats, and to derive better algorithms for
  training over sparse-quantized formats.
paper_ai_summary: A study on scaling laws and compression techniques shows that a unified capacity metric can predict model performance across different compressed formats, including scalar-quantized, sparse-quantized, and vector-quantized representations.
paper_ai_keywords: scaling laws model compression quantization sparsification sparse representations scalar-quantized sparse-quantized vector-quantized parameter efficiency Gaussian data capacity metric