title: MixLLM: LLM Quantization with Global Mixed-precision between
    Output-features and Highly-efficient System Design
summary: Quantization has become one of the most effective methodologies to compress
  LLMs into smaller size. However, the existing quantization solutions still show
  limitations of either non-negligible accuracy drop or system inefficiency. In
  this paper, we make a comprehensive analysis of the general quantization
  principles on their effect to the triangle of accuracy, memory consumption and
  system efficiency. We propose MixLLM that explores the new optimization space
  of mixed-precision quantization between output features based on the insight
  that different output features matter differently in the model. MixLLM
  identifies the output features with high salience in the global view rather
  than within each single layer, effectively assigning the larger bit-width to
  output features that need it most to achieve good accuracy with low memory
  consumption. We present the sweet spot of quantization configuration of
  algorithm-system co-design that leads to high accuracy and system efficiency.
  To address the system challenge, we design the two-step dequantization to make
  use of the int8 Tensor Core easily and fast data type conversion to reduce
  dequantization overhead significantly, and present the software pipeline to
  overlap the memory access, dequantization and the MatMul to the best. Extensive
  experiments show that with only 10% more bits, the PPL increasement can be
  reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while on
  average MMLU-Pro improves by 0.93 over the SOTA of three popular models. In
  addition to its superior accuracy, MixLLM also achieves state-of-the-art system
  efficiency.
paper_authors._id: 676827d8bc5af30a79857669
paper_authors.name: Zhen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:10:47.069Z
paper_authors._id: 65373b2c89dd48faca859d02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
paper_authors.fullname: Zhen Zheng
paper_authors.type: user
paper_authors.user: JamesTheZ
paper_authors._id: 676827d8bc5af30a7985766a
paper_authors.name: Xiaonan Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:34:23.206Z
paper_authors._id: 66839c29d88415938469b483
paper_authors.avatarUrl: /avatars/38af0856922ea53f607a7158e640923c.svg
paper_authors.fullname: Xiaonan Song
paper_authors.type: user
paper_authors.user: xiaonans
paper_authors._id: 676827d8bc5af30a7985766b
paper_authors.name: Chuanjie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:34:28.717Z
paper_authors._id: 6528b76d0b44378dcc54ec60
paper_authors.avatarUrl: /avatars/0943b4c7def02c1751e163aa758ccec5.svg
paper_authors.fullname: Chuanjie Liu
paper_authors.type: user
paper_authors.user: chuanjieliu
paper_title: MixLLM: LLM Quantization with Global Mixed-precision between
    Output-features and Highly-efficient System Design
paper_summary: Quantization has become one of the most effective methodologies to compress
  LLMs into smaller size. However, the existing quantization solutions still show
  limitations of either non-negligible accuracy drop or system inefficiency. In
  this paper, we make a comprehensive analysis of the general quantization
  principles on their effect to the triangle of accuracy, memory consumption and
  system efficiency. We propose MixLLM that explores the new optimization space
  of mixed-precision quantization between output features based on the insight
  that different output features matter differently in the model. MixLLM
  identifies the output features with high salience in the global view rather
  than within each single layer, effectively assigning the larger bit-width to
  output features that need it most to achieve good accuracy with low memory
  consumption. We present the sweet spot of quantization configuration of
  algorithm-system co-design that leads to high accuracy and system efficiency.
  To address the system challenge, we design the two-step dequantization to make
  use of the int8 Tensor Core easily and fast data type conversion to reduce
  dequantization overhead significantly, and present the software pipeline to
  overlap the memory access, dequantization and the MatMul to the best. Extensive
  experiments show that with only 10% more bits, the PPL increasement can be
  reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while on
  average MMLU-Pro improves by 0.93 over the SOTA of three popular models. In
  addition to its superior accuracy, MixLLM also achieves state-of-the-art system
  efficiency.
paper_ai_summary: MixLLM, a mixed-precision quantization technique, enhances the accuracy and efficiency of large language models by optimizing output feature quantization and system-level dequantization processes.
paper_ai_keywords: quantization mixed-precision quantization output features salience bit-width dequantization Tensor Core MatMul PPL MMLU-Pro system efficiency algorithm-system co-design