title: BitNet: Scaling 1-bit Transformers for Large Language Models
summary: The increasing size of large language models has posed challenges for
  deployment and raised concerns about environmental impact due to high energy
  consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
  Transformer architecture designed for large language models. Specifically, we
  introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
  train 1-bit weights from scratch. Experimental results on language modeling
  show that BitNet achieves competitive performance while substantially reducing
  memory footprint and energy consumption, compared to state-of-the-art 8-bit
  quantization methods and FP16 Transformer baselines. Furthermore, BitNet
  exhibits a scaling law akin to full-precision Transformers, suggesting its
  potential for effective scaling to even larger language models while
  maintaining efficiency and performance benefits.
paper_authors._id: 652f50c3a0f516a99df4c569
paper_authors.name: Hongyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-18T16:50:05.910Z
paper_authors._id: 63f71771d36951307fcb4dcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
paper_authors.fullname: Hongyu Wang
paper_authors.type: user
paper_authors.user: hongyuw
paper_authors._id: 652f50c3a0f516a99df4c56a
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T08:10:20.044Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 652f50c3a0f516a99df4c56b
paper_authors.name: Li Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-16T09:50:18.468Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 652f50c3a0f516a99df4c56c
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T08:11:17.453Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 652f50c3a0f516a99df4c56d
paper_authors.name: Huaijie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T08:11:25.055Z
paper_authors._id: 6310664063b70252b4779150
paper_authors.avatarUrl: /avatars/d514270c57b6ac494e0a419d792a72e5.svg
paper_authors.fullname: Huaijie Wang
paper_authors.type: user
paper_authors.user: jwhj
paper_authors._id: 652f50c3a0f516a99df4c56e
paper_authors.name: Lingxiao Ma
paper_authors._id: 652f50c3a0f516a99df4c56f
paper_authors.name: Fan Yang
paper_authors._id: 652f50c3a0f516a99df4c570
paper_authors.name: Ruiping Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T08:13:49.707Z
paper_authors._id: 64fa7e742d20ced4e9ecd835
paper_authors.avatarUrl: /avatars/52db257f8600562905be61d16b78a4f4.svg
paper_authors.fullname: ruipingwang
paper_authors.type: user
paper_authors.user: RuipingWang1986
paper_authors._id: 652f50c3a0f516a99df4c571
paper_authors.name: Yi Wu
paper_authors._id: 652f50c3a0f516a99df4c572
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T08:14:30.689Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: BitNet: Scaling 1-bit Transformers for Large Language Models
paper_summary: The increasing size of large language models has posed challenges for
  deployment and raised concerns about environmental impact due to high energy
  consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
  Transformer architecture designed for large language models. Specifically, we
  introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
  train 1-bit weights from scratch. Experimental results on language modeling
  show that BitNet achieves competitive performance while substantially reducing
  memory footprint and energy consumption, compared to state-of-the-art 8-bit
  quantization methods and FP16 Transformer baselines. Furthermore, BitNet
  exhibits a scaling law akin to full-precision Transformers, suggesting its
  potential for effective scaling to even larger language models while
  maintaining efficiency and performance benefits.
paper_ai_summary: BitNet, a 1-bit Transformer architecture, reduces memory and energy consumption while achieving competitive performance in language modeling.
paper_ai_keywords: BitNet 1-bit Transformer BitLinear nn.Linear 1-bit weights memory footprint energy consumption 8-bit quantization FP16 scaling law full-precision Transformers