Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for
  Quantized LLMs with 100T Training Tokens
We reveal that low-bit quantization favors undertrained large language models
(LLMs) by observing that models with larger sizes or fewer training tokens
experience less quantization-induced degradation (QiD) when applying low-bit
quantization, whereas smaller models with extensive training tokens suffer
significant QiD. To gain deeper insights into this trend, we study over 1500
quantized LLM checkpoints of various sizes and at different training levels
(undertrained or fully trained) in a controlled setting, deriving scaling laws
for understanding the relationship between QiD and factors such as the number
of training tokens, model size and bit width.
  With the derived scaling laws, we propose a novel perspective that we can use
QiD to measure an LLM's training levels and determine the number of training
tokens required for fully training LLMs of various sizes. Moreover, we use the
scaling laws to predict the quantization performance of different-sized LLMs
trained with 100 trillion tokens. Our projection shows that the low-bit
quantization performance of future models, which are expected to be trained
with over 100 trillion tokens, may NOT be desirable. This poses a potential
challenge for low-bit quantization in the future and highlights the need for
awareness of a model's training level when evaluating low-bit quantization
research. To facilitate future research on this problem, we release all the
1500+ quantized checkpoints used in this work at
https://huggingface.co/Xu-Ouyang.
67469174db412d1e84f9d8be
Xu Ouyang
67469174db412d1e84f9d8bf
Tao Ge
claimed_verified
2024-11-27T22:20:14.890Z
64ab5db78a1a9187c215f45a
/avatars/17add4cd02e183f9b9d4feb7b6c27c21.svg
Tao Ge
user
sggetao
67469174db412d1e84f9d8c0
Thomas Hartvigsen
67469174db412d1e84f9d8c1
Zhisong Zhang
67469174db412d1e84f9d8c2
Haitao Mi
67469174db412d1e84f9d8c3
Dong Yu
Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for
  Quantized LLMs with 100T Training Tokens
We reveal that low-bit quantization favors undertrained large language models
(LLMs) by observing that models with larger sizes or fewer training tokens
experience less quantization-induced degradation (QiD) when applying low-bit
quantization, whereas smaller models with extensive training tokens suffer
significant QiD. To gain deeper insights into this trend, we study over 1500
quantized LLM checkpoints of various sizes and at different training levels
(undertrained or fully trained) in a controlled setting, deriving scaling laws
for understanding the relationship between QiD and factors such as the number
of training tokens, model size and bit width.
  With the derived scaling laws, we propose a novel perspective that we can use
QiD to measure an LLM's training levels and determine the number of training
tokens required for fully training LLMs of various sizes. Moreover, we use the
scaling laws to predict the quantization performance of different-sized LLMs
trained with 100 trillion tokens. Our projection shows that the low-bit
quantization performance of future models, which are expected to be trained
with over 100 trillion tokens, may NOT be desirable. This poses a potential
challenge for low-bit quantization in the future and highlights the need for
awareness of a model's training level when evaluating low-bit quantization
research. To facilitate future research on this problem, we release all the
1500+ quantized checkpoints used in this work at
https://huggingface.co/Xu-Ouyang.
Low-bit quantization is more effective for undertrained large language models and can be used to assess the training level of models, but may struggle with future larger, fully trained models.
low-bit quantization quantization-induced degradation (QiD) large language models (LLMs) quantized LLM checkpoints scaling laws undertrained fully trained bit width