title: BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
summary: Pretrained large language models (LLMs) exhibit exceptional general language
  processing capabilities but come with significant demands on memory and
  computational resources. As a powerful compression technology, binarization can
  extremely reduce model weights to a mere 1 bit, lowering the expensive
  computation and memory requirements. However, existing quantization techniques
  fall short of maintaining LLM performance under ultra-low bit-widths. In
  response to this challenge, we present BiLLM, a groundbreaking 1-bit
  post-training quantization scheme tailored for pretrained LLMs. Based on the
  weight distribution of LLMs, BiLLM first identifies and structurally selects
  salient weights, and minimizes the compression loss through an effective binary
  residual approximation strategy. Moreover, considering the bell-shaped
  distribution of the non-salient weights, we propose an optimal splitting search
  to group and binarize them accurately. BiLLM achieving for the first time
  high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit
  weights across various LLMs families and evaluation metrics, outperforms SOTA
  quantization methods of LLM by significant margins. Moreover, BiLLM enables the
  binarization process of the LLM with 7 billion weights within 0.5 hours on a
  single GPU, demonstrating satisfactory time efficiency.
paper_authors._id: 65c43a4ff19b126a3cd92d10
paper_authors.name: Wei Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:55:21.980Z
paper_authors._id: 656db3f53dc1d277e5a64410
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
paper_authors.fullname: Wei Huang
paper_authors.type: user
paper_authors.user: AaronHuangWei
paper_authors._id: 65c43a4ff19b126a3cd92d11
paper_authors.name: Yangdong Liu
paper_authors._id: 65c43a4ff19b126a3cd92d12
paper_authors.name: Haotong Qin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-03T00:46:32.543Z
paper_authors._id: 65c49589c0b1921e19260a8d
paper_authors.avatarUrl: /avatars/7ce9af8c627f2a0c3db6bde82290ee1f.svg
paper_authors.fullname: Haotong Qin
paper_authors.type: user
paper_authors.user: HaotongQin
paper_authors._id: 65c43a4ff19b126a3cd92d13
paper_authors.name: Ying Li
paper_authors._id: 65c43a4ff19b126a3cd92d14
paper_authors.name: Shiming Zhang
paper_authors._id: 65c43a4ff19b126a3cd92d15
paper_authors.name: Xianglong Liu
paper_authors._id: 65c43a4ff19b126a3cd92d16
paper_authors.name: Michele Magno
paper_authors._id: 65c43a4ff19b126a3cd92d17
paper_authors.name: Xiaojuan Qi
paper_title: BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
paper_summary: Pretrained large language models (LLMs) exhibit exceptional general language
  processing capabilities but come with significant demands on memory and
  computational resources. As a powerful compression technology, binarization can
  extremely reduce model weights to a mere 1 bit, lowering the expensive
  computation and memory requirements. However, existing quantization techniques
  fall short of maintaining LLM performance under ultra-low bit-widths. In
  response to this challenge, we present BiLLM, a groundbreaking 1-bit
  post-training quantization scheme tailored for pretrained LLMs. Based on the
  weight distribution of LLMs, BiLLM first identifies and structurally selects
  salient weights, and minimizes the compression loss through an effective binary
  residual approximation strategy. Moreover, considering the bell-shaped
  distribution of the non-salient weights, we propose an optimal splitting search
  to group and binarize them accurately. BiLLM achieving for the first time
  high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit
  weights across various LLMs families and evaluation metrics, outperforms SOTA
  quantization methods of LLM by significant margins. Moreover, BiLLM enables the
  binarization process of the LLM with 7 billion weights within 0.5 hours on a
  single GPU, demonstrating satisfactory time efficiency.
paper_ai_summary: BiLLM, a 1-bit post-training quantization scheme for pretrained LLMs, achieves high-accuracy inference with reduced computational and memory requirements.
paper_ai_keywords: binarization post-training quantization salient weights binary residual approximation optimal splitting search bell-shaped distribution perplexity LLaMA2-70B SOTA