BitNet v2: Native 4-bit Activations with Hadamard Transformation for
  1-bit LLMs
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by
activation outliers, which complicate quantization to low bit-widths. We
introduce BitNet v2, a novel framework enabling native 4-bit activation
quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward
network activations, we propose H-BitLinear, a module applying an online
Hadamard transformation prior to activation quantization. This transformation
smooths sharp activation distributions into more Gaussian-like forms, suitable
for low-bit representation. Experiments show BitNet v2 trained from scratch
with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2
achieves minimal performance degradation when trained with native 4-bit
activations, significantly reducing memory footprint and computational cost for
batched inference.
680ef1549cc294f617fb14b4
Hongyu Wang
admin_assigned
2025-04-29T12:31:53.757Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
680ef1549cc294f617fb14b5
Shuming Ma
admin_assigned
2025-04-29T12:31:38.000Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
680ef1549cc294f617fb14b6
Furu Wei
admin_assigned
2025-04-29T12:31:32.056Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
BitNet v2: Native 4-bit Activations with Hadamard Transformation for
  1-bit LLMs
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by
activation outliers, which complicate quantization to low bit-widths. We
introduce BitNet v2, a novel framework enabling native 4-bit activation
quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward
network activations, we propose H-BitLinear, a module applying an online
Hadamard transformation prior to activation quantization. This transformation
smooths sharp activation distributions into more Gaussian-like forms, suitable
for low-bit representation. Experiments show BitNet v2 trained from scratch
with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2
achieves minimal performance degradation when trained with native 4-bit
activations, significantly reducing memory footprint and computational cost for
batched inference.
BitNet v2 enables native 4-bit activation quantization for 1-bit Large Language Models by using an online Hadamard transformation to mitigate activation outliers, achieving performance comparable to 8-bit activations with reduced memory and computational costs.
Large Language Models LLMS BitNet H-BitLinear Hadamard transformation activation quantization activation outliers attention network feed-forward network