title: ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric
    Strategy for Diverse Generative Tasks
summary: This study examines 4-bit quantization methods like GPTQ in large language
  models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
  Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
  extend task scope to more generative categories such as code generation and
  abstractive summarization, in which we found that INT4 quantization can
  significantly underperform. However, simply shifting to higher precision
  formats like FP6 has been particularly challenging, thus overlooked, due to
  poor performance caused by the lack of sophisticated integration and system
  acceleration strategies on current AI hardware. Our results show that FP6, even
  with a coarse-grain quantization scheme, performs robustly across various
  algorithms and tasks, demonstrating its superiority in accuracy and
  versatility. Notably, with the FP6 quantization, \codestar-15B model performs
  comparably to its FP16 counterpart in code generation, and for smaller models
  like the 406M it closely matches their baselines in summarization. Neither can
  be achieved by INT4. To better accommodate various AI hardware and achieve the
  best system performance, we propose a novel 4+2 design for FP6 to achieve
  similar latency to the state-of-the-art INT4 fine-grain quantization. With our
  design, FP6 can become a promising solution to the current 4-bit quantization
  methods used in LLMs.
paper_authors._id: 657bc7f3696ec3dda92fc12e
paper_authors.name: Xiaoxia Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:00:05.048Z
paper_authors._id: 619721da1726b360d54d9ff7
paper_authors.avatarUrl: /avatars/f5f8bf0f372bef58848530dd7d7f58de.svg
paper_authors.fullname: wu
paper_authors.type: user
paper_authors.user: xiaoxia-microsoft
paper_authors._id: 657bc7f3696ec3dda92fc12f
paper_authors.name: Haojun Xia
paper_authors._id: 657bc7f3696ec3dda92fc130
paper_authors.name: Stephen Youn
paper_authors._id: 657bc7f3696ec3dda92fc131
paper_authors.name: Zhen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-26T14:12:07.834Z
paper_authors._id: 65373b2c89dd48faca859d02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
paper_authors.fullname: Zhen Zheng
paper_authors.type: user
paper_authors.user: JamesTheZ
paper_authors._id: 657bc7f3696ec3dda92fc132
paper_authors.name: Shiyang Chen
paper_authors._id: 657bc7f3696ec3dda92fc133
paper_authors.name: Arash Bakhtiari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-15T19:10:40.497Z
paper_authors._id: 6553cda0c81411e2aaf930b1
paper_authors.avatarUrl: /avatars/291e4e9cef092d649ab734a3f09a3af1.svg
paper_authors.fullname: Arash Bakhtiari
paper_authors.type: user
paper_authors.user: arashb
paper_authors._id: 657bc7f3696ec3dda92fc134
paper_authors.name: Michael Wyatt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:01:35.693Z
paper_authors._id: 62910c52d1be2630c0d38a72
paper_authors.avatarUrl: /avatars/09f3e2646a506b5aeb57558eaa769046.svg
paper_authors.fullname: Michael Wyatt
paper_authors.type: user
paper_authors.user: mwyatt
paper_authors._id: 657bc7f3696ec3dda92fc135
paper_authors.name: Yuxiong He
paper_authors._id: 657bc7f3696ec3dda92fc136
paper_authors.name: Olatunji Ruwase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:01:52.108Z
paper_authors._id: 6499d685ea2cdac80992e742
paper_authors.avatarUrl: /avatars/c32741e7fc57c9a08722fab3877a7b81.svg
paper_authors.fullname: Olatunji Ruwase
paper_authors.type: user
paper_authors.user: tjruwase
paper_authors._id: 657bc7f3696ec3dda92fc137
paper_authors.name: Leon Song
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-15T03:28:52.104Z
paper_authors._id: 651442f5c75a3d4c44de4ac9
paper_authors.avatarUrl: /avatars/c60185389bf6379e4360cb54615922ed.svg
paper_authors.fullname: Shuaiwen Leon Song
paper_authors.type: user
paper_authors.user: leonangel991
paper_authors._id: 657bc7f3696ec3dda92fc138
paper_authors.name: Zhewei Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:01:59.162Z
paper_authors._id: 6352e1953199c2612ec2ff01
paper_authors.avatarUrl: /avatars/f75444f1b7cba63ed085774a950a3fc7.svg
paper_authors.fullname: Zhewei Yao
paper_authors.type: user
paper_authors.user: zheweiyao
paper_title: ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric
    Strategy for Diverse Generative Tasks
paper_summary: This study examines 4-bit quantization methods like GPTQ in large language
  models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
  Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
  extend task scope to more generative categories such as code generation and
  abstractive summarization, in which we found that INT4 quantization can
  significantly underperform. However, simply shifting to higher precision
  formats like FP6 has been particularly challenging, thus overlooked, due to
  poor performance caused by the lack of sophisticated integration and system
  acceleration strategies on current AI hardware. Our results show that FP6, even
  with a coarse-grain quantization scheme, performs robustly across various
  algorithms and tasks, demonstrating its superiority in accuracy and
  versatility. Notably, with the FP6 quantization, \codestar-15B model performs
  comparably to its FP16 counterpart in code generation, and for smaller models
  like the 406M it closely matches their baselines in summarization. Neither can
  be achieved by INT4. To better accommodate various AI hardware and achieve the
  best system performance, we propose a novel 4+2 design for FP6 to achieve
  similar latency to the state-of-the-art INT4 fine-grain quantization. With our
  design, FP6 can become a promising solution to the current 4-bit quantization
  methods used in LLMs.
paper_ai_keywords: GPTQ Zero-Shot tasks code generation abstractive summarization quantization INT4 FP6 Zero-Shot tasks coarse-grain quantization latency fine-grain quantization LLMs