title: DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and
    DeepSpeed-Inference
summary: The deployment and scaling of large language models (LLMs) have become
  critical as they permeate various applications, demanding high-throughput and
  low-latency serving systems. Existing frameworks struggle to balance these
  requirements, especially for workloads with long prompts. This paper introduces
  DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and
  generation composition strategy, to deliver up to 2.3x higher effective
  throughput, 2x lower latency on average, and up to 3.7x lower (token-level)
  tail latency, compared to state-of-the-art systems like vLLM. We leverage a
  synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an
  efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced
  implementation supports a range of models and offers both non-persistent and
  persistent deployment options, catering to diverse user scenarios from
  interactive sessions to long-running applications. We present a detailed
  benchmarking methodology, analyze the performance through latency-throughput
  curves, and investigate scalability via load balancing. Our evaluations
  demonstrate substantial improvements in throughput and latency across various
  models and hardware configurations. We discuss our roadmap for future
  enhancements, including broader model support and new hardware backends. The
  DeepSpeed-FastGen code is readily available for community engagement and
  contribution.
paper_authors._id: 65a896d39aec16459940d5fd
paper_authors.name: Connor Holmes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:20:21.356Z
paper_authors._id: 630e826cc0eca3037a04e9c7
paper_authors.avatarUrl: /avatars/d71f42635f3e678acae4eb99cf9d8431.svg
paper_authors.fullname: Connor Holmes
paper_authors.type: user
paper_authors.user: cmikeh2
paper_authors._id: 65a896d39aec16459940d5fe
paper_authors.name: Masahiro Tanaka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:20:40.022Z
paper_authors._id: 65160d2c8a8269300db0a3b5
paper_authors.avatarUrl: /avatars/176d34200eca72791ad7f8393917513d.svg
paper_authors.fullname: Masahiro Tanaka
paper_authors.type: user
paper_authors.user: mtanaka
paper_authors._id: 65a896d39aec16459940d5ff
paper_authors.name: Michael Wyatt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:20:54.201Z
paper_authors._id: 62910c52d1be2630c0d38a72
paper_authors.avatarUrl: /avatars/40a72a10e3ea17809a09bde77360c194.svg
paper_authors.fullname: Michael Wyatt
paper_authors.type: user
paper_authors.user: mwyatt
paper_authors._id: 65a896d39aec16459940d600
paper_authors.name: Ammar Ahmad Awan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:21:21.108Z
paper_authors._id: 6324b8417d6d0cbbe2c65019
paper_authors.avatarUrl: /avatars/7c1a468daaa984203a600a3b3526298f.svg
paper_authors.fullname: Ammar Awan
paper_authors.type: user
paper_authors.user: ammarawan
paper_authors._id: 65a896d39aec16459940d601
paper_authors.name: Jeff Rasley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:21:27.489Z
paper_authors._id: 60871172511e863acdb20b9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60871172511e863acdb20b9f/qlhLl8hAryhQY3uvjoZ3i.jpeg
paper_authors.fullname: Jeff Rasley
paper_authors.type: user
paper_authors.user: jeffra
paper_authors._id: 65a896d39aec16459940d602
paper_authors.name: Samyam Rajbhandari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:21:33.367Z
paper_authors._id: 63238e79d444e1d90888ea03
paper_authors.avatarUrl: /avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
paper_authors.fullname: Samyam Rajbhandari
paper_authors.type: user
paper_authors.user: samyam
paper_authors._id: 65a896d39aec16459940d603
paper_authors.name: Reza Yazdani Aminabadi
paper_authors._id: 65a896d39aec16459940d604
paper_authors.name: Heyang Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:21:47.716Z
paper_authors._id: 63bc6a611374e3ef9134a24e
paper_authors.avatarUrl: /avatars/9df037b9cf7d7952a10d919e24a2ffa2.svg
paper_authors.fullname: Heyang Qin
paper_authors.type: user
paper_authors.user: heyangqin
paper_authors._id: 65a896d39aec16459940d605
paper_authors.name: Arash Bakhtiari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:26:26.348Z
paper_authors._id: 6553cda0c81411e2aaf930b1
paper_authors.avatarUrl: /avatars/291e4e9cef092d649ab734a3f09a3af1.svg
paper_authors.fullname: Arash Bakhtiari
paper_authors.type: user
paper_authors.user: arashb
paper_authors._id: 65a896d39aec16459940d606
paper_authors.name: Lev Kurilenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:22:24.973Z
paper_authors._id: 635852141d66b442317cbe26
paper_authors.avatarUrl: /avatars/bcf9164f6ac6393ffecc5f4383d12cbe.svg
paper_authors.fullname: Lev Kurilenko
paper_authors.type: user
paper_authors.user: lekurile
paper_authors._id: 65a896d39aec16459940d607
paper_authors.name: Yuxiong He
paper_title: DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and
    DeepSpeed-Inference
paper_summary: The deployment and scaling of large language models (LLMs) have become
  critical as they permeate various applications, demanding high-throughput and
  low-latency serving systems. Existing frameworks struggle to balance these
  requirements, especially for workloads with long prompts. This paper introduces
  DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and
  generation composition strategy, to deliver up to 2.3x higher effective
  throughput, 2x lower latency on average, and up to 3.7x lower (token-level)
  tail latency, compared to state-of-the-art systems like vLLM. We leverage a
  synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an
  efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced
  implementation supports a range of models and offers both non-persistent and
  persistent deployment options, catering to diverse user scenarios from
  interactive sessions to long-running applications. We present a detailed
  benchmarking methodology, analyze the performance through latency-throughput
  curves, and investigate scalability via load balancing. Our evaluations
  demonstrate substantial improvements in throughput and latency across various
  models and hardware configurations. We discuss our roadmap for future
  enhancements, including broader model support and new hardware backends. The
  DeepSpeed-FastGen code is readily available for community engagement and
  contribution.
paper_ai_summary: DeepSpeed-FastGen enhances the deployment of large language models by introducing Dynamic SplitFuse, improving throughput and latency compared to existing frameworks.
paper_ai_keywords: DeepSpeed-FastGen Dynamic SplitFuse DeepSpeed-MII DeepSpeed-Inference prompt and generation composition non-persistent deployment persistent deployment latency-throughput curves load balancing hardware backends