Infinite-LLM: Efficient LLM Service for Long Context with DistAttention
  and Distributed KVCache
The rapid proliferation of Large Language Models (LLMs) has been a driving
force in the growth of cloud-based LLM services, which are now integral to
advancing AI applications. However, the dynamic auto-regressive nature of LLM
service, along with the need to support exceptionally long context lengths,
demands the flexible allocation and release of substantial resources. This
presents considerable challenges in designing cloud-based LLM service systems,
where inefficient management can lead to performance degradation or resource
wastage. In response to these challenges, this work introduces DistAttention, a
novel distributed attention algorithm that segments the KV Cache into smaller,
manageable units, enabling distributed processing and storage of the attention
module. Based on that, we propose DistKV-LLM, a distributed LLM serving system
that dynamically manages KV Cache and effectively orchestrates all accessible
GPU and CPU memories spanning across the data center. This ensures a
high-performance LLM service on the cloud, adaptable to a broad range of
context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in
configurations from 2 to 32 instances, our system exhibited 1.03-2.4x
end-to-end throughput improvements and supported context lengths 2-19x longer
than current state-of-the-art LLM service systems, as evidenced by extensive
testing across 18 datasets with context lengths up to 1,900K.
659b5a56c6457161ca21a9f3
Bin Lin
659b5a56c6457161ca21a9f4
Tao Peng
659b5a56c6457161ca21a9f5
Chen Zhang
659b5a56c6457161ca21a9f6
Minmin Sun
admin_assigned
2024-01-08T11:48:39.984Z
636a390037d9329b4a007009
/avatars/a3c9117e104d4667e39e20ec83dc5cd6.svg
Minmin Sun
user
minminsun
659b5a56c6457161ca21a9f7
Lanbo Li
659b5a56c6457161ca21a9f8
Hanyu Zhao
admin_assigned
2024-01-08T11:49:14.935Z
64ddd52063c1d48544ece154
/avatars/000134fdc04dc822705bcabb1ef631ad.svg
HANYU ZHAO
user
jerryzhao0122
659b5a56c6457161ca21a9f9
Wencong Xiao
659b5a56c6457161ca21a9fa
Qi Xu
659b5a56c6457161ca21a9fb
Xiafei Qiu
admin_assigned
2024-01-08T11:49:45.817Z
6312d35017e4bd0d77ee3798
/avatars/55640fcb7f3c5effcf7d73924dceef1b.svg
xiafei
user
qiuxiafei
659b5a56c6457161ca21a9fc
Shen Li
659b5a56c6457161ca21a9fd
Zhigang Ji
659b5a56c6457161ca21a9fe
Yong Li
659b5a56c6457161ca21a9ff
Wei Lin
Infinite-LLM: Efficient LLM Service for Long Context with DistAttention
  and Distributed KVCache
The rapid proliferation of Large Language Models (LLMs) has been a driving
force in the growth of cloud-based LLM services, which are now integral to
advancing AI applications. However, the dynamic auto-regressive nature of LLM
service, along with the need to support exceptionally long context lengths,
demands the flexible allocation and release of substantial resources. This
presents considerable challenges in designing cloud-based LLM service systems,
where inefficient management can lead to performance degradation or resource
wastage. In response to these challenges, this work introduces DistAttention, a
novel distributed attention algorithm that segments the KV Cache into smaller,
manageable units, enabling distributed processing and storage of the attention
module. Based on that, we propose DistKV-LLM, a distributed LLM serving system
that dynamically manages KV Cache and effectively orchestrates all accessible
GPU and CPU memories spanning across the data center. This ensures a
high-performance LLM service on the cloud, adaptable to a broad range of
context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in
configurations from 2 to 32 instances, our system exhibited 1.03-2.4x
end-to-end throughput improvements and supported context lengths 2-19x longer
than current state-of-the-art LLM service systems, as evidenced by extensive
testing across 18 datasets with context lengths up to 1,900K.
DistKV-LLM, a distributed LLM serving system using DistAttention, improves cloud-based LLM performance and context length handling through optimized resource management.
Large Language Models LLMs LLM service dynamic auto-regressive nature long context lengths distributed attention algorithm KV Cache DistAttention DistKV-LLM GPU CPU data center end-to-end throughput context lengths