title: Infinite-LLM: Efficient LLM Service for Long Context with DistAttention
    and Distributed KVCache
summary: The rapid proliferation of Large Language Models (LLMs) has been a driving
  force in the growth of cloud-based LLM services, which are now integral to
  advancing AI applications. However, the dynamic auto-regressive nature of LLM
  service, along with the need to support exceptionally long context lengths,
  demands the flexible allocation and release of substantial resources. This
  presents considerable challenges in designing cloud-based LLM service systems,
  where inefficient management can lead to performance degradation or resource
  wastage. In response to these challenges, this work introduces DistAttention, a
  novel distributed attention algorithm that segments the KV Cache into smaller,
  manageable units, enabling distributed processing and storage of the attention
  module. Based on that, we propose DistKV-LLM, a distributed LLM serving system
  that dynamically manages KV Cache and effectively orchestrates all accessible
  GPU and CPU memories spanning across the data center. This ensures a
  high-performance LLM service on the cloud, adaptable to a broad range of
  context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in
  configurations from 2 to 32 instances, our system exhibited 1.03-2.4x
  end-to-end throughput improvements and supported context lengths 2-19x longer
  than current state-of-the-art LLM service systems, as evidenced by extensive
  testing across 18 datasets with context lengths up to 1,900K.
paper_authors._id: 659b5a56c6457161ca21a9f3
paper_authors.name: Bin Lin
paper_authors._id: 659b5a56c6457161ca21a9f4
paper_authors.name: Tao Peng
paper_authors._id: 659b5a56c6457161ca21a9f5
paper_authors.name: Chen Zhang
paper_authors._id: 659b5a56c6457161ca21a9f6
paper_authors.name: Minmin Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:48:39.984Z
paper_authors._id: 636a390037d9329b4a007009
paper_authors.avatarUrl: /avatars/a3c9117e104d4667e39e20ec83dc5cd6.svg
paper_authors.fullname: Minmin Sun
paper_authors.type: user
paper_authors.user: minminsun
paper_authors._id: 659b5a56c6457161ca21a9f7
paper_authors.name: Lanbo Li
paper_authors._id: 659b5a56c6457161ca21a9f8
paper_authors.name: Hanyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:49:14.935Z
paper_authors._id: 64ddd52063c1d48544ece154
paper_authors.avatarUrl: /avatars/000134fdc04dc822705bcabb1ef631ad.svg
paper_authors.fullname: HANYU ZHAO
paper_authors.type: user
paper_authors.user: jerryzhao0122
paper_authors._id: 659b5a56c6457161ca21a9f9
paper_authors.name: Wencong Xiao
paper_authors._id: 659b5a56c6457161ca21a9fa
paper_authors.name: Qi Xu
paper_authors._id: 659b5a56c6457161ca21a9fb
paper_authors.name: Xiafei Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:49:45.817Z
paper_authors._id: 6312d35017e4bd0d77ee3798
paper_authors.avatarUrl: /avatars/55640fcb7f3c5effcf7d73924dceef1b.svg
paper_authors.fullname: xiafei
paper_authors.type: user
paper_authors.user: qiuxiafei
paper_authors._id: 659b5a56c6457161ca21a9fc
paper_authors.name: Shen Li
paper_authors._id: 659b5a56c6457161ca21a9fd
paper_authors.name: Zhigang Ji
paper_authors._id: 659b5a56c6457161ca21a9fe
paper_authors.name: Yong Li
paper_authors._id: 659b5a56c6457161ca21a9ff
paper_authors.name: Wei Lin
paper_title: Infinite-LLM: Efficient LLM Service for Long Context with DistAttention
    and Distributed KVCache
paper_summary: The rapid proliferation of Large Language Models (LLMs) has been a driving
  force in the growth of cloud-based LLM services, which are now integral to
  advancing AI applications. However, the dynamic auto-regressive nature of LLM
  service, along with the need to support exceptionally long context lengths,
  demands the flexible allocation and release of substantial resources. This
  presents considerable challenges in designing cloud-based LLM service systems,
  where inefficient management can lead to performance degradation or resource
  wastage. In response to these challenges, this work introduces DistAttention, a
  novel distributed attention algorithm that segments the KV Cache into smaller,
  manageable units, enabling distributed processing and storage of the attention
  module. Based on that, we propose DistKV-LLM, a distributed LLM serving system
  that dynamically manages KV Cache and effectively orchestrates all accessible
  GPU and CPU memories spanning across the data center. This ensures a
  high-performance LLM service on the cloud, adaptable to a broad range of
  context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in
  configurations from 2 to 32 instances, our system exhibited 1.03-2.4x
  end-to-end throughput improvements and supported context lengths 2-19x longer
  than current state-of-the-art LLM service systems, as evidenced by extensive
  testing across 18 datasets with context lengths up to 1,900K.
paper_ai_summary: DistKV-LLM, a distributed LLM serving system using DistAttention, improves cloud-based LLM performance and context length handling through optimized resource management.
paper_ai_keywords: Large Language Models LLMs LLM service dynamic auto-regressive nature long context lengths distributed attention algorithm KV Cache DistAttention DistKV-LLM GPU CPU data center end-to-end throughput context lengths