title: BitNet v2: Native 4-bit Activations with Hadamard Transformation for
    1-bit LLMs
summary: Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by
  activation outliers, which complicate quantization to low bit-widths. We
  introduce BitNet v2, a novel framework enabling native 4-bit activation
  quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward
  network activations, we propose H-BitLinear, a module applying an online
  Hadamard transformation prior to activation quantization. This transformation
  smooths sharp activation distributions into more Gaussian-like forms, suitable
  for low-bit representation. Experiments show BitNet v2 trained from scratch
  with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2
  achieves minimal performance degradation when trained with native 4-bit
  activations, significantly reducing memory footprint and computational cost for
  batched inference.
paper_authors._id: 680ef1549cc294f617fb14b4
paper_authors.name: Hongyu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:31:53.757Z
paper_authors._id: 63f71771d36951307fcb4dcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
paper_authors.fullname: Hongyu Wang
paper_authors.type: user
paper_authors.user: hongyuw
paper_authors._id: 680ef1549cc294f617fb14b5
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:31:38.000Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 680ef1549cc294f617fb14b6
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:31:32.056Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: BitNet v2: Native 4-bit Activations with Hadamard Transformation for
    1-bit LLMs
paper_summary: Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by
  activation outliers, which complicate quantization to low bit-widths. We
  introduce BitNet v2, a novel framework enabling native 4-bit activation
  quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward
  network activations, we propose H-BitLinear, a module applying an online
  Hadamard transformation prior to activation quantization. This transformation
  smooths sharp activation distributions into more Gaussian-like forms, suitable
  for low-bit representation. Experiments show BitNet v2 trained from scratch
  with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2
  achieves minimal performance degradation when trained with native 4-bit
  activations, significantly reducing memory footprint and computational cost for
  batched inference.
paper_ai_summary: BitNet v2 enables native 4-bit activation quantization for 1-bit Large Language Models by using an online Hadamard transformation to mitigate activation outliers, achieving performance comparable to 8-bit activations with reduced memory and computational costs.
paper_ai_keywords: Large Language Models LLMS BitNet H-BitLinear Hadamard transformation activation quantization activation outliers attention network feed-forward network