DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and
  DeepSpeed-Inference
The deployment and scaling of large language models (LLMs) have become
critical as they permeate various applications, demanding high-throughput and
low-latency serving systems. Existing frameworks struggle to balance these
requirements, especially for workloads with long prompts. This paper introduces
DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and
generation composition strategy, to deliver up to 2.3x higher effective
throughput, 2x lower latency on average, and up to 3.7x lower (token-level)
tail latency, compared to state-of-the-art systems like vLLM. We leverage a
synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an
efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced
implementation supports a range of models and offers both non-persistent and
persistent deployment options, catering to diverse user scenarios from
interactive sessions to long-running applications. We present a detailed
benchmarking methodology, analyze the performance through latency-throughput
curves, and investigate scalability via load balancing. Our evaluations
demonstrate substantial improvements in throughput and latency across various
models and hardware configurations. We discuss our roadmap for future
enhancements, including broader model support and new hardware backends. The
DeepSpeed-FastGen code is readily available for community engagement and
contribution.
65a896d39aec16459940d5fd
Connor Holmes
admin_assigned
2024-01-18T09:20:21.356Z
630e826cc0eca3037a04e9c7
/avatars/d71f42635f3e678acae4eb99cf9d8431.svg
Connor Holmes
user
cmikeh2
65a896d39aec16459940d5fe
Masahiro Tanaka
admin_assigned
2024-01-18T09:20:40.022Z
65160d2c8a8269300db0a3b5
/avatars/176d34200eca72791ad7f8393917513d.svg
Masahiro Tanaka
user
mtanaka
65a896d39aec16459940d5ff
Michael Wyatt
admin_assigned
2024-01-18T09:20:54.201Z
62910c52d1be2630c0d38a72
/avatars/40a72a10e3ea17809a09bde77360c194.svg
Michael Wyatt
user
mwyatt
65a896d39aec16459940d600
Ammar Ahmad Awan
admin_assigned
2024-01-18T09:21:21.108Z
6324b8417d6d0cbbe2c65019
/avatars/7c1a468daaa984203a600a3b3526298f.svg
Ammar Awan
user
ammarawan
65a896d39aec16459940d601
Jeff Rasley
admin_assigned
2024-01-18T09:21:27.489Z
60871172511e863acdb20b9f
https://cdn-avatars.huggingface.co/v1/production/uploads/60871172511e863acdb20b9f/qlhLl8hAryhQY3uvjoZ3i.jpeg
Jeff Rasley
user
jeffra
65a896d39aec16459940d602
Samyam Rajbhandari
admin_assigned
2024-01-18T09:21:33.367Z
63238e79d444e1d90888ea03
/avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
Samyam Rajbhandari
user
samyam
65a896d39aec16459940d603
Reza Yazdani Aminabadi
65a896d39aec16459940d604
Heyang Qin
admin_assigned
2024-01-18T09:21:47.716Z
63bc6a611374e3ef9134a24e
/avatars/9df037b9cf7d7952a10d919e24a2ffa2.svg
Heyang Qin
user
heyangqin
65a896d39aec16459940d605
Arash Bakhtiari
claimed_verified
2024-01-29T08:26:26.348Z
6553cda0c81411e2aaf930b1
/avatars/291e4e9cef092d649ab734a3f09a3af1.svg
Arash Bakhtiari
user
arashb
65a896d39aec16459940d606
Lev Kurilenko
admin_assigned
2024-01-18T09:22:24.973Z
635852141d66b442317cbe26
/avatars/bcf9164f6ac6393ffecc5f4383d12cbe.svg
Lev Kurilenko
user
lekurile
65a896d39aec16459940d607
Yuxiong He
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and
  DeepSpeed-Inference
The deployment and scaling of large language models (LLMs) have become
critical as they permeate various applications, demanding high-throughput and
low-latency serving systems. Existing frameworks struggle to balance these
requirements, especially for workloads with long prompts. This paper introduces
DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and
generation composition strategy, to deliver up to 2.3x higher effective
throughput, 2x lower latency on average, and up to 3.7x lower (token-level)
tail latency, compared to state-of-the-art systems like vLLM. We leverage a
synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an
efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced
implementation supports a range of models and offers both non-persistent and
persistent deployment options, catering to diverse user scenarios from
interactive sessions to long-running applications. We present a detailed
benchmarking methodology, analyze the performance through latency-throughput
curves, and investigate scalability via load balancing. Our evaluations
demonstrate substantial improvements in throughput and latency across various
models and hardware configurations. We discuss our roadmap for future
enhancements, including broader model support and new hardware backends. The
DeepSpeed-FastGen code is readily available for community engagement and
contribution.
DeepSpeed-FastGen enhances the deployment of large language models by introducing Dynamic SplitFuse, improving throughput and latency compared to existing frameworks.
DeepSpeed-FastGen Dynamic SplitFuse DeepSpeed-MII DeepSpeed-Inference prompt and generation composition non-persistent deployment persistent deployment latency-throughput curves load balancing hardware backends