title: OneBit: Towards Extremely Low-bit Large Language Models
summary: Model quantification uses low bit-width values to represent the weight
  matrices of models, which is a promising approach to reduce both storage and
  computational overheads of deploying highly anticipated LLMs. However, existing
  quantization methods suffer severe performance degradation when the bit-width
  is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
  quantize models. This paper boldly quantizes the weight matrices of LLMs to
  1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
  this target, we introduce a 1-bit quantization-aware training (QAT) framework
  named OneBit, including a novel 1-bit parameter representation method to better
  quantize LLMs as well as an effective parameter initialization method based on
  matrix decomposition to improve the convergence speed of the QAT framework.
  Sufficient experimental results indicate that OneBit achieves good performance
  (at least 83% of the non-quantized performance) with robust training processes
  when only using 1-bit weight matrices.
paper_authors._id: 65d43c4d9f087171b8fa0474
paper_authors.name: Yuzhuang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:01:49.435Z
paper_authors._id: 64b89a5403124195cd840b91
paper_authors.avatarUrl: /avatars/4141144d4df2ca652931c8c15bd24b7c.svg
paper_authors.fullname: Yuzhuang Xu
paper_authors.type: user
paper_authors.user: Yuzhuang
paper_authors._id: 65d43c4d9f087171b8fa0475
paper_authors.name: Xu Han
paper_authors._id: 65d43c4d9f087171b8fa0476
paper_authors.name: Zonghan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:32.346Z
paper_authors._id: 635795f24fba057662cdf2e7
paper_authors.avatarUrl: /avatars/07642c7cfd9b57a0f8382268b009cf6f.svg
paper_authors.fullname: Zonghan Yang
paper_authors.type: user
paper_authors.user: minicheshire
paper_authors._id: 65d43c4d9f087171b8fa0477
paper_authors.name: Shuo Wang
paper_authors._id: 65d43c4d9f087171b8fa0478
paper_authors.name: Qingfu Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:03:26.373Z
paper_authors._id: 649ce641dd87dd9ef76e3628
paper_authors.avatarUrl: /avatars/d5fdeb9e2131ada258b776fec95eb00b.svg
paper_authors.fullname: zhu
paper_authors.type: user
paper_authors.user: qingfuhf
paper_authors._id: 65d43c4d9f087171b8fa0479
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T08:00:50.567Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 65d43c4d9f087171b8fa047a
paper_authors.name: Weidong Liu
paper_authors._id: 65d43c4d9f087171b8fa047b
paper_authors.name: Wanxiang Che
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:05:05.609Z
paper_authors._id: 6325418ae5d49a0d40f66176
paper_authors.avatarUrl: /avatars/59a0e29ce09e9113edc53cebb728b099.svg
paper_authors.fullname: Wanxiang Che
paper_authors.type: user
paper_authors.user: wanxiangche
paper_title: OneBit: Towards Extremely Low-bit Large Language Models
paper_summary: Model quantification uses low bit-width values to represent the weight
  matrices of models, which is a promising approach to reduce both storage and
  computational overheads of deploying highly anticipated LLMs. However, existing
  quantization methods suffer severe performance degradation when the bit-width
  is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
  quantize models. This paper boldly quantizes the weight matrices of LLMs to
  1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
  this target, we introduce a 1-bit quantization-aware training (QAT) framework
  named OneBit, including a novel 1-bit parameter representation method to better
  quantize LLMs as well as an effective parameter initialization method based on
  matrix decomposition to improve the convergence speed of the QAT framework.
  Sufficient experimental results indicate that OneBit achieves good performance
  (at least 83% of the non-quantized performance) with robust training processes
  when only using 1-bit weight matrices.
paper_ai_summary: OneBit, a 1-bit quantization-aware training framework, allows LLMs to achieve robust performance with 1-bit weight matrices, reducing storage and computational overhead.
paper_ai_keywords: model quantification low bit-width weight matrices LLMs quantization methods 1-bit quantization-aware training (QAT) parameter representation matrix decomposition performance convergence speed