BitNet: Scaling 1-bit Transformers for Large Language Models
The increasing size of large language models has posed challenges for
deployment and raised concerns about environmental impact due to high energy
consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
Transformer architecture designed for large language models. Specifically, we
introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
train 1-bit weights from scratch. Experimental results on language modeling
show that BitNet achieves competitive performance while substantially reducing
memory footprint and energy consumption, compared to state-of-the-art 8-bit
quantization methods and FP16 Transformer baselines. Furthermore, BitNet
exhibits a scaling law akin to full-precision Transformers, suggesting its
potential for effective scaling to even larger language models while
maintaining efficiency and performance benefits.
652f50c3a0f516a99df4c569
Hongyu Wang
claimed_verified
2023-10-18T16:50:05.910Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
652f50c3a0f516a99df4c56a
Shuming Ma
admin_assigned
2023-10-18T08:10:20.044Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
652f50c3a0f516a99df4c56b
Li Dong
admin_assigned
2024-10-16T09:50:18.468Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
652f50c3a0f516a99df4c56c
Shaohan Huang
admin_assigned
2023-10-18T08:11:17.453Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
652f50c3a0f516a99df4c56d
Huaijie Wang
admin_assigned
2023-10-18T08:11:25.055Z
6310664063b70252b4779150
/avatars/d514270c57b6ac494e0a419d792a72e5.svg
Huaijie Wang
user
jwhj
652f50c3a0f516a99df4c56e
Lingxiao Ma
652f50c3a0f516a99df4c56f
Fan Yang
652f50c3a0f516a99df4c570
Ruiping Wang
admin_assigned
2023-10-18T08:13:49.707Z
64fa7e742d20ced4e9ecd835
/avatars/52db257f8600562905be61d16b78a4f4.svg
ruipingwang
user
RuipingWang1986
652f50c3a0f516a99df4c571
Yi Wu
652f50c3a0f516a99df4c572
Furu Wei
admin_assigned
2023-10-18T08:14:30.689Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
BitNet: Scaling 1-bit Transformers for Large Language Models
The increasing size of large language models has posed challenges for
deployment and raised concerns about environmental impact due to high energy
consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
Transformer architecture designed for large language models. Specifically, we
introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
train 1-bit weights from scratch. Experimental results on language modeling
show that BitNet achieves competitive performance while substantially reducing
memory footprint and energy consumption, compared to state-of-the-art 8-bit
quantization methods and FP16 Transformer baselines. Furthermore, BitNet
exhibits a scaling law akin to full-precision Transformers, suggesting its
potential for effective scaling to even larger language models while
maintaining efficiency and performance benefits.
BitNet, a 1-bit Transformer architecture, reduces memory and energy consumption while achieving competitive performance in language modeling.
BitNet 1-bit Transformer BitLinear nn.Linear 1-bit weights memory footprint energy consumption 8-bit quantization FP16 scaling law full-precision Transformers