title: BitNet a4.8: 4-bit Activations for 1-bit LLMs
summary: Recent research on the 1-bit Large Language Models (LLMs), such as BitNet
  b1.58, presents a promising direction for reducing the inference cost of LLMs
  while maintaining their performance. In this work, we introduce BitNet a4.8,
  enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid
  quantization and sparsification strategy to mitigate the quantization errors
  introduced by the outlier channels. Specifically, we utilize 4-bit activations
  for inputs to the attention and feed-forward network layers, while sparsifying
  intermediate states followed with 8-bit quantization. Extensive experiments
  demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58
  with equivalent training costs, while being faster in inference with enabling
  4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of
  parameters and supports 3-bit KV cache, further enhancing the efficiency of
  large-scale LLM deployment and inference.
paper_authors._id: 672daab4c4c5c4fce10738fc
paper_authors.name: Hongyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:25.856Z
paper_authors._id: 63f71771d36951307fcb4dcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
paper_authors.fullname: Hongyu Wang
paper_authors.type: user
paper_authors.user: hongyuw
paper_authors._id: 672daab4c4c5c4fce10738fd
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:27:15.986Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 672daab4c4c5c4fce10738fe
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:27:09.585Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: BitNet a4.8: 4-bit Activations for 1-bit LLMs
paper_summary: Recent research on the 1-bit Large Language Models (LLMs), such as BitNet
  b1.58, presents a promising direction for reducing the inference cost of LLMs
  while maintaining their performance. In this work, we introduce BitNet a4.8,
  enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid
  quantization and sparsification strategy to mitigate the quantization errors
  introduced by the outlier channels. Specifically, we utilize 4-bit activations
  for inputs to the attention and feed-forward network layers, while sparsifying
  intermediate states followed with 8-bit quantization. Extensive experiments
  demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58
  with equivalent training costs, while being faster in inference with enabling
  4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of
  parameters and supports 3-bit KV cache, further enhancing the efficiency of
  large-scale LLM deployment and inference.
paper_ai_keywords: hybrid quantization sparsification quantization errors outlier channels 4-bit activations attention layers feed-forward network layers 8-bit quantization intermediate states 4-bit kernels (INT4/FP4) parameter-efficient fine-tuning 3-bit KV cache