Unified Scaling Laws for Compressed Representations
Scaling laws have shaped recent advances in machine learning by enabling
predictable scaling of model performance based on model size, computation, and
data volume. Concurrently, the rise in computational cost for AI has motivated
model compression techniques, notably quantization and sparsification, which
have emerged to mitigate the steep computational demands associated with
large-scale training and inference. This paper investigates the interplay
between scaling laws and compression formats, exploring whether a unified
scaling framework can accurately predict model performance when training occurs
over various compressed representations, such as sparse, scalar-quantized,
sparse-quantized or even vector-quantized formats. Our key contributions
include validating a general scaling law formulation and showing that it is
applicable both individually but also composably across compression types.
Based on this, our main finding is demonstrating both theoretically and
empirically that there exists a simple "capacity" metric -- based on the
representation's ability to fit random Gaussian data -- which can robustly
predict parameter efficiency across multiple compressed representations. On the
practical side, we extend our formulation to directly compare the accuracy
potential of different compressed formats, and to derive better algorithms for
training over sparse-quantized formats.
683ee1a6e65df2618201e81b
Andrei Panferov
683ee1a6e65df2618201e81c
Alexandra Volkova
claimed_verified
2025-06-03T12:33:09.324Z
65f08bd773f2e1e291244d86
/avatars/4d66171163094024ebc82bb59859d9c5.svg
Alexandra
user
alexandraww
683ee1a6e65df2618201e81d
Ionut-Vlad Modoranu
683ee1a6e65df2618201e81e
Vage Egiazarian
683ee1a6e65df2618201e81f
Mher Safaryan
683ee1a6e65df2618201e820
Dan Alistarh
Unified Scaling Laws for Compressed Representations
Scaling laws have shaped recent advances in machine learning by enabling
predictable scaling of model performance based on model size, computation, and
data volume. Concurrently, the rise in computational cost for AI has motivated
model compression techniques, notably quantization and sparsification, which
have emerged to mitigate the steep computational demands associated with
large-scale training and inference. This paper investigates the interplay
between scaling laws and compression formats, exploring whether a unified
scaling framework can accurately predict model performance when training occurs
over various compressed representations, such as sparse, scalar-quantized,
sparse-quantized or even vector-quantized formats. Our key contributions
include validating a general scaling law formulation and showing that it is
applicable both individually but also composably across compression types.
Based on this, our main finding is demonstrating both theoretically and
empirically that there exists a simple "capacity" metric -- based on the
representation's ability to fit random Gaussian data -- which can robustly
predict parameter efficiency across multiple compressed representations. On the
practical side, we extend our formulation to directly compare the accuracy
potential of different compressed formats, and to derive better algorithms for
training over sparse-quantized formats.
A study on scaling laws and compression techniques shows that a unified capacity metric can predict model performance across different compressed formats, including scalar-quantized, sparse-quantized, and vector-quantized representations.
scaling laws model compression quantization sparsification sparse representations scalar-quantized sparse-quantized vector-quantized parameter efficiency Gaussian data capacity metric