FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric
  Algorithm-System Co-Design
Six-bit quantization (FP6) can effectively reduce the size of large language
models (LLMs) and preserve the model quality consistently across varied
applications. However, existing systems do not provide Tensor Core support for
FP6 quantization and struggle to achieve practical performance improvements
during LLM inference. It is challenging to support FP6 quantization on GPUs due
to (1) unfriendly memory access of model weights with irregular bit-width and
(2) high runtime overhead of weight de-quantization. To address these problems,
we propose TC-FPx, the first full-stack GPU kernel design scheme with unified
Tensor Core support of float-point weights for various quantization bit-width.
We integrate TC-FPx kernel into an existing inference system, providing new
end-to-end support (called FP6-LLM) for quantized LLM inference, where better
trade-offs between inference cost and model quality are achieved. Experiments
show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU,
achieving 1.69x-2.65x higher normalized inference throughput than the FP16
baseline. The source code will be publicly available soon.
65b31f8fe312613c603d2c38
Haojun Xia
admin_assigned
2024-01-26T10:18:27.303Z
658287b3f7111631de53d33d
/avatars/bb679fe7e9a16afc569093c9c76639fa.svg
Haojun Xia
user
xhjustc
65b31f8fe312613c603d2c39
Zhen Zheng
admin_assigned
2024-01-26T10:18:35.044Z
65373b2c89dd48faca859d02
https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
Zhen Zheng
user
JamesTheZ
65b31f8fe312613c603d2c3a
Xiaoxia Wu
65b31f8fe312613c603d2c3b
Shiyang Chen
admin_assigned
2024-01-26T10:18:54.538Z
65838eb1cbb381e78871e11e
/avatars/490072573daa0c8833b07a89f27242ed.svg
Shiyang Chen
user
cctry-ru
65b31f8fe312613c603d2c3c
Zhewei Yao
admin_assigned
2024-01-26T10:19:01.508Z
6352e1953199c2612ec2ff01
/avatars/f75444f1b7cba63ed085774a950a3fc7.svg
Zhewei Yao
user
zheweiyao
65b31f8fe312613c603d2c3d
Stephen Youn
65b31f8fe312613c603d2c3e
Arash Bakhtiari
claimed_verified
2024-01-29T08:26:14.244Z
6553cda0c81411e2aaf930b1
/avatars/291e4e9cef092d649ab734a3f09a3af1.svg
Arash Bakhtiari
user
arashb
65b31f8fe312613c603d2c3f
Michael Wyatt
admin_assigned
2024-01-26T10:19:58.246Z
62910c52d1be2630c0d38a72
/avatars/40a72a10e3ea17809a09bde77360c194.svg
Michael Wyatt
user
mwyatt
65b31f8fe312613c603d2c40
Donglin Zhuang
admin_assigned
2024-01-26T10:20:08.239Z
618a58e1d128cf918549eff5
/avatars/f8520ebfef221633051f39f83085d7b4.svg
Zhuang
user
Donglin
65b31f8fe312613c603d2c41
Zhongzhu Zhou
admin_assigned
2024-01-26T10:20:18.574Z
6426781213a9e5d9675a1d57
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vR7u0azaShh5Tj_OqdP2k.png
ZHONGZHU ZHOU
user
Zhongzhu
65b31f8fe312613c603d2c42
Olatunji Ruwase
admin_assigned
2024-01-26T10:20:24.948Z
6499d685ea2cdac80992e742
/avatars/c32741e7fc57c9a08722fab3877a7b81.svg
Olatunji Ruwase
user
tjruwase
65b31f8fe312613c603d2c43
Yuxiong He
65b31f8fe312613c603d2c44
Shuaiwen Leon Song
admin_assigned
2024-01-26T10:20:39.695Z
651442f5c75a3d4c44de4ac9
/avatars/c60185389bf6379e4360cb54615922ed.svg
Shuaiwen Leon Song
user
leonangel991
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric
  Algorithm-System Co-Design
Six-bit quantization (FP6) can effectively reduce the size of large language
models (LLMs) and preserve the model quality consistently across varied
applications. However, existing systems do not provide Tensor Core support for
FP6 quantization and struggle to achieve practical performance improvements
during LLM inference. It is challenging to support FP6 quantization on GPUs due
to (1) unfriendly memory access of model weights with irregular bit-width and
(2) high runtime overhead of weight de-quantization. To address these problems,
we propose TC-FPx, the first full-stack GPU kernel design scheme with unified
Tensor Core support of float-point weights for various quantization bit-width.
We integrate TC-FPx kernel into an existing inference system, providing new
end-to-end support (called FP6-LLM) for quantized LLM inference, where better
trade-offs between inference cost and model quality are achieved. Experiments
show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU,
achieving 1.69x-2.65x higher normalized inference throughput than the FP16
baseline. The source code will be publicly available soon.
TC-FPx, a full-stack GPU kernel design, enables efficient six-bit quantization of large language models with Tensor Core support, achieving higher inference throughput on a single GPU.
Six-bit quantization FP6 large language models LLMs Tensor Core FP6 quantization TC-FPx FP6-LLM LLaMA-70b normalized inference throughput FP16