title: Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for
    Quantized LLMs with 100T Training Tokens
summary: We reveal that low-bit quantization favors undertrained large language models
  (LLMs) by observing that models with larger sizes or fewer training tokens
  experience less quantization-induced degradation (QiD) when applying low-bit
  quantization, whereas smaller models with extensive training tokens suffer
  significant QiD. To gain deeper insights into this trend, we study over 1500
  quantized LLM checkpoints of various sizes and at different training levels
  (undertrained or fully trained) in a controlled setting, deriving scaling laws
  for understanding the relationship between QiD and factors such as the number
  of training tokens, model size and bit width.
    With the derived scaling laws, we propose a novel perspective that we can use
  QiD to measure an LLM's training levels and determine the number of training
  tokens required for fully training LLMs of various sizes. Moreover, we use the
  scaling laws to predict the quantization performance of different-sized LLMs
  trained with 100 trillion tokens. Our projection shows that the low-bit
  quantization performance of future models, which are expected to be trained
  with over 100 trillion tokens, may NOT be desirable. This poses a potential
  challenge for low-bit quantization in the future and highlights the need for
  awareness of a model's training level when evaluating low-bit quantization
  research. To facilitate future research on this problem, we release all the
  1500+ quantized checkpoints used in this work at
  https://huggingface.co/Xu-Ouyang.
paper_authors._id: 67469174db412d1e84f9d8be
paper_authors.name: Xu Ouyang
paper_authors._id: 67469174db412d1e84f9d8bf
paper_authors.name: Tao Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:14.890Z
paper_authors._id: 64ab5db78a1a9187c215f45a
paper_authors.avatarUrl: /avatars/17add4cd02e183f9b9d4feb7b6c27c21.svg
paper_authors.fullname: Tao Ge
paper_authors.type: user
paper_authors.user: sggetao
paper_authors._id: 67469174db412d1e84f9d8c0
paper_authors.name: Thomas Hartvigsen
paper_authors._id: 67469174db412d1e84f9d8c1
paper_authors.name: Zhisong Zhang
paper_authors._id: 67469174db412d1e84f9d8c2
paper_authors.name: Haitao Mi
paper_authors._id: 67469174db412d1e84f9d8c3
paper_authors.name: Dong Yu
paper_title: Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for
    Quantized LLMs with 100T Training Tokens
paper_summary: We reveal that low-bit quantization favors undertrained large language models
  (LLMs) by observing that models with larger sizes or fewer training tokens
  experience less quantization-induced degradation (QiD) when applying low-bit
  quantization, whereas smaller models with extensive training tokens suffer
  significant QiD. To gain deeper insights into this trend, we study over 1500
  quantized LLM checkpoints of various sizes and at different training levels
  (undertrained or fully trained) in a controlled setting, deriving scaling laws
  for understanding the relationship between QiD and factors such as the number
  of training tokens, model size and bit width.
    With the derived scaling laws, we propose a novel perspective that we can use
  QiD to measure an LLM's training levels and determine the number of training
  tokens required for fully training LLMs of various sizes. Moreover, we use the
  scaling laws to predict the quantization performance of different-sized LLMs
  trained with 100 trillion tokens. Our projection shows that the low-bit
  quantization performance of future models, which are expected to be trained
  with over 100 trillion tokens, may NOT be desirable. This poses a potential
  challenge for low-bit quantization in the future and highlights the need for
  awareness of a model's training level when evaluating low-bit quantization
  research. To facilitate future research on this problem, we release all the
  1500+ quantized checkpoints used in this work at
  https://huggingface.co/Xu-Ouyang.
paper_ai_keywords: low-bit quantization undertrained large language models (LLMs) quantization-induced degradation (QiD) scaling laws quantization performance model size bit width training tokens fully trained LLMs