title: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
summary: Recent research, such as BitNet, is paving the way for a new era of 1-bit
  Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant,
  namely BitNet b1.58, in which every single parameter (or weight) of the LLM is
  ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16)
  Transformer LLM with the same model size and training tokens in terms of both
  perplexity and end-task performance, while being significantly more
  cost-effective in terms of latency, memory, throughput, and energy consumption.
  More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for
  training new generations of LLMs that are both high-performance and
  cost-effective. Furthermore, it enables a new computation paradigm and opens
  the door for designing specific hardware optimized for 1-bit LLMs.
paper_authors._id: 65ded4ff33599459308af370
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T11:01:44.539Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 65ded4ff33599459308af371
paper_authors.name: Hongyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-28T11:41:14.755Z
paper_authors._id: 63f71771d36951307fcb4dcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
paper_authors.fullname: Hongyu Wang
paper_authors.type: user
paper_authors.user: hongyuw
paper_authors._id: 65ded4ff33599459308af372
paper_authors.name: Lingxiao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T13:16:16.361Z
paper_authors._id: 63d9074f89a9683b46d1c73d
paper_authors.avatarUrl: /avatars/6dc070c7618f24b76d9df9882fcd5aed.svg
paper_authors.fullname: Lingxiao Ma
paper_authors.type: user
paper_authors.user: xysmlx
paper_authors._id: 65ded4ff33599459308af373
paper_authors.name: Lei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:42:25.603Z
paper_authors._id: 646c7b3849a37754bdeefe8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646c7b3849a37754bdeefe8e/tejirZfCHm6EqhXOdnMUq.jpeg
paper_authors.fullname: Lei
paper_authors.type: user
paper_authors.user: LeiWang1999
paper_authors._id: 65ded4ff33599459308af374
paper_authors.name: Wenhui Wang
paper_authors._id: 65ded4ff33599459308af375
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T11:06:04.566Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 65ded4ff33599459308af376
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T15:53:48.354Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 65ded4ff33599459308af377
paper_authors.name: Ruiping Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T11:05:07.170Z
paper_authors._id: 64fa7e742d20ced4e9ecd835
paper_authors.avatarUrl: /avatars/52db257f8600562905be61d16b78a4f4.svg
paper_authors.fullname: ruipingwang
paper_authors.type: user
paper_authors.user: RuipingWang1986
paper_authors._id: 65ded4ff33599459308af378
paper_authors.name: Jilong Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T11:04:47.786Z
paper_authors._id: 632bfc0e4a4991e7114f2a83
paper_authors.avatarUrl: /avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
paper_authors.fullname: Jilong Xue
paper_authors.type: user
paper_authors.user: jlxue
paper_authors._id: 65ded4ff33599459308af379
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T11:03:55.843Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
paper_summary: Recent research, such as BitNet, is paving the way for a new era of 1-bit
  Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant,
  namely BitNet b1.58, in which every single parameter (or weight) of the LLM is
  ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16)
  Transformer LLM with the same model size and training tokens in terms of both
  perplexity and end-task performance, while being significantly more
  cost-effective in terms of latency, memory, throughput, and energy consumption.
  More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for
  training new generations of LLMs that are both high-performance and
  cost-effective. Furthermore, it enables a new computation paradigm and opens
  the door for designing specific hardware optimized for 1-bit LLMs.
paper_ai_summary: A 1-bit LLM variant, BitNet b1.58, achieves comparable performance to full-precision models with reduced computational costs and introduces new scaling laws and hardware design opportunities.
paper_ai_keywords: BitNet 1-bit LLM ternary Transformer LLM perplexity end-task performance latency memory throughput energy consumption scaling law hardware optimization