title: QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
summary: Mixture-of-Experts (MoE) architectures offer a general solution to the high
  inference costs of large language models (LLMs) via sparse routing, bringing
  faster and more accurate models, at the cost of massive parameter counts. For
  example, the SwitchTransformer-c2048 model has 1.6 trillion parameters,
  requiring 3.2TB of accelerator memory to run efficiently, which makes practical
  deployment challenging and expensive. In this paper, we present a solution to
  this memory problem, in form of a new compression and execution framework
  called QMoE. Specifically, QMoE consists of a scalable algorithm which
  accurately compresses trillion-parameter MoEs to less than 1 bit per parameter,
  in a custom format co-designed with bespoke GPU decoding kernels to facilitate
  efficient end-to-end compressed inference, with minor runtime overheads
  relative to uncompressed execution. Concretely, QMoE can compress the 1.6
  trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x
  compression, 0.8 bits per parameter) at only minor accuracy loss, in less than
  a day on a single GPU. This enables, for the first time, the execution of a
  trillion-parameter model on affordable commodity hardware, like a single server
  with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead
  relative to ideal uncompressed inference. The source code and compressed models
  are available at github.com/IST-DASLab/qmoe.
paper_authors._id: 6539deec690022ca51f016e7
paper_authors.name: Elias Frantar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:33:44.712Z
paper_authors._id: 64b79f81037d6452a317b846
paper_authors.avatarUrl: /avatars/c9c9770136804f8e47f52ff1a4cd120a.svg
paper_authors.fullname: Elias Frantar
paper_authors.type: user
paper_authors.user: efrantar
paper_authors._id: 6539deec690022ca51f016e8
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:33:54.454Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_title: QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
paper_summary: Mixture-of-Experts (MoE) architectures offer a general solution to the high
  inference costs of large language models (LLMs) via sparse routing, bringing
  faster and more accurate models, at the cost of massive parameter counts. For
  example, the SwitchTransformer-c2048 model has 1.6 trillion parameters,
  requiring 3.2TB of accelerator memory to run efficiently, which makes practical
  deployment challenging and expensive. In this paper, we present a solution to
  this memory problem, in form of a new compression and execution framework
  called QMoE. Specifically, QMoE consists of a scalable algorithm which
  accurately compresses trillion-parameter MoEs to less than 1 bit per parameter,
  in a custom format co-designed with bespoke GPU decoding kernels to facilitate
  efficient end-to-end compressed inference, with minor runtime overheads
  relative to uncompressed execution. Concretely, QMoE can compress the 1.6
  trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x
  compression, 0.8 bits per parameter) at only minor accuracy loss, in less than
  a day on a single GPU. This enables, for the first time, the execution of a
  trillion-parameter model on affordable commodity hardware, like a single server
  with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead
  relative to ideal uncompressed inference. The source code and compressed models
  are available at github.com/IST-DASLab/qmoe.
paper_ai_keywords: Mixture-of-Experts (MoE) architectures sparse routing large language models (LLMs) SwitchTransformer-c2048 parameter counts accelerator memory QMoE compression and execution framework bespoke GPU decoding kernels end-to-end compressed inference runtime overheads uncompressed execution commodity hardware NVIDIA A6000 NVIDIA 3090 GPUs accuracy loss