title: FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric
    Algorithm-System Co-Design
summary: Six-bit quantization (FP6) can effectively reduce the size of large language
  models (LLMs) and preserve the model quality consistently across varied
  applications. However, existing systems do not provide Tensor Core support for
  FP6 quantization and struggle to achieve practical performance improvements
  during LLM inference. It is challenging to support FP6 quantization on GPUs due
  to (1) unfriendly memory access of model weights with irregular bit-width and
  (2) high runtime overhead of weight de-quantization. To address these problems,
  we propose TC-FPx, the first full-stack GPU kernel design scheme with unified
  Tensor Core support of float-point weights for various quantization bit-width.
  We integrate TC-FPx kernel into an existing inference system, providing new
  end-to-end support (called FP6-LLM) for quantized LLM inference, where better
  trade-offs between inference cost and model quality are achieved. Experiments
  show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU,
  achieving 1.69x-2.65x higher normalized inference throughput than the FP16
  baseline. The source code will be publicly available soon.
paper_authors._id: 65b31f8fe312613c603d2c38
paper_authors.name: Haojun Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:18:27.303Z
paper_authors._id: 658287b3f7111631de53d33d
paper_authors.avatarUrl: /avatars/bb679fe7e9a16afc569093c9c76639fa.svg
paper_authors.fullname: Haojun Xia
paper_authors.type: user
paper_authors.user: xhjustc
paper_authors._id: 65b31f8fe312613c603d2c39
paper_authors.name: Zhen Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:18:35.044Z
paper_authors._id: 65373b2c89dd48faca859d02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
paper_authors.fullname: Zhen Zheng
paper_authors.type: user
paper_authors.user: JamesTheZ
paper_authors._id: 65b31f8fe312613c603d2c3a
paper_authors.name: Xiaoxia Wu
paper_authors._id: 65b31f8fe312613c603d2c3b
paper_authors.name: Shiyang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:18:54.538Z
paper_authors._id: 65838eb1cbb381e78871e11e
paper_authors.avatarUrl: /avatars/490072573daa0c8833b07a89f27242ed.svg
paper_authors.fullname: Shiyang Chen
paper_authors.type: user
paper_authors.user: cctry-ru
paper_authors._id: 65b31f8fe312613c603d2c3c
paper_authors.name: Zhewei Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:19:01.508Z
paper_authors._id: 6352e1953199c2612ec2ff01
paper_authors.avatarUrl: /avatars/f75444f1b7cba63ed085774a950a3fc7.svg
paper_authors.fullname: Zhewei Yao
paper_authors.type: user
paper_authors.user: zheweiyao
paper_authors._id: 65b31f8fe312613c603d2c3d
paper_authors.name: Stephen Youn
paper_authors._id: 65b31f8fe312613c603d2c3e
paper_authors.name: Arash Bakhtiari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:26:14.244Z
paper_authors._id: 6553cda0c81411e2aaf930b1
paper_authors.avatarUrl: /avatars/291e4e9cef092d649ab734a3f09a3af1.svg
paper_authors.fullname: Arash Bakhtiari
paper_authors.type: user
paper_authors.user: arashb
paper_authors._id: 65b31f8fe312613c603d2c3f
paper_authors.name: Michael Wyatt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:19:58.246Z
paper_authors._id: 62910c52d1be2630c0d38a72
paper_authors.avatarUrl: /avatars/40a72a10e3ea17809a09bde77360c194.svg
paper_authors.fullname: Michael Wyatt
paper_authors.type: user
paper_authors.user: mwyatt
paper_authors._id: 65b31f8fe312613c603d2c40
paper_authors.name: Donglin Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:20:08.239Z
paper_authors._id: 618a58e1d128cf918549eff5
paper_authors.avatarUrl: /avatars/f8520ebfef221633051f39f83085d7b4.svg
paper_authors.fullname: Zhuang
paper_authors.type: user
paper_authors.user: Donglin
paper_authors._id: 65b31f8fe312613c603d2c41
paper_authors.name: Zhongzhu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:20:18.574Z
paper_authors._id: 6426781213a9e5d9675a1d57
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vR7u0azaShh5Tj_OqdP2k.png
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Zhongzhu
paper_authors._id: 65b31f8fe312613c603d2c42
paper_authors.name: Olatunji Ruwase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:20:24.948Z
paper_authors._id: 6499d685ea2cdac80992e742
paper_authors.avatarUrl: /avatars/c32741e7fc57c9a08722fab3877a7b81.svg
paper_authors.fullname: Olatunji Ruwase
paper_authors.type: user
paper_authors.user: tjruwase
paper_authors._id: 65b31f8fe312613c603d2c43
paper_authors.name: Yuxiong He
paper_authors._id: 65b31f8fe312613c603d2c44
paper_authors.name: Shuaiwen Leon Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:20:39.695Z
paper_authors._id: 651442f5c75a3d4c44de4ac9
paper_authors.avatarUrl: /avatars/c60185389bf6379e4360cb54615922ed.svg
paper_authors.fullname: Shuaiwen Leon Song
paper_authors.type: user
paper_authors.user: leonangel991
paper_title: FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric
    Algorithm-System Co-Design
paper_summary: Six-bit quantization (FP6) can effectively reduce the size of large language
  models (LLMs) and preserve the model quality consistently across varied
  applications. However, existing systems do not provide Tensor Core support for
  FP6 quantization and struggle to achieve practical performance improvements
  during LLM inference. It is challenging to support FP6 quantization on GPUs due
  to (1) unfriendly memory access of model weights with irregular bit-width and
  (2) high runtime overhead of weight de-quantization. To address these problems,
  we propose TC-FPx, the first full-stack GPU kernel design scheme with unified
  Tensor Core support of float-point weights for various quantization bit-width.
  We integrate TC-FPx kernel into an existing inference system, providing new
  end-to-end support (called FP6-LLM) for quantized LLM inference, where better
  trade-offs between inference cost and model quality are achieved. Experiments
  show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU,
  achieving 1.69x-2.65x higher normalized inference throughput than the FP16
  baseline. The source code will be publicly available soon.
paper_ai_summary: TC-FPx, a full-stack GPU kernel design, enables efficient six-bit quantization of large language models with Tensor Core support, achieving higher inference throughput on a single GPU.
paper_ai_keywords: Six-bit quantization FP6 large language models LLMs Tensor Core FP6 quantization TC-FPx FP6-LLM LLaMA-70b normalized inference throughput FP16