Taming the Titans: A Survey of Efficient LLM Inference Serving
Large Language Models (LLMs) for Generative AI have achieved remarkable
progress, evolving into sophisticated and versatile tools widely adopted across
various domains and applications. However, the substantial memory overhead
caused by their vast number of parameters, combined with the high computational
demands of the attention mechanism, poses significant challenges in achieving
low latency and high throughput for LLM inference services. Recent
advancements, driven by groundbreaking research, have significantly accelerated
progress in this field. This paper provides a comprehensive survey of these
methods, covering fundamental instance-level approaches, in-depth cluster-level
strategies, emerging scenario directions, and other miscellaneous but important
areas. At the instance level, we review model placement, request scheduling,
decoding length prediction, storage management, and the disaggregation
paradigm. At the cluster level, we explore GPU cluster deployment,
multi-instance load balancing, and cloud service solutions. For emerging
scenarios, we organize the discussion around specific tasks, modules, and
auxiliary methods. To ensure a holistic overview, we also highlight several
niche yet critical areas. Finally, we outline potential research directions to
further advance the field of LLM inference serving.
68118b63439785d7ff81a879
Ranran Zhen
claimed_verified
2025-04-30T07:56:15.143Z
647e96507f9ad5e44bac50bd
/avatars/2551f99401540676cfe5cd0ed99e70c1.svg
Ranran Zhen
user
zenRRan
68118b63439785d7ff81a87a
Juntao Li
admin_assigned
2025-05-05T08:12:36.237Z
6670e285b0c03c4e9d6e0985
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
Juntao Li
user
douvleplus
68118b63439785d7ff81a87b
Yixin Ji
68118b63439785d7ff81a87c
Zhenlin Yang
admin_assigned
2025-05-05T08:13:47.806Z
64023a3dcad36eb2b0f9c2c0
/avatars/306ce26d991ec58da89ccec37922d906.svg
Zhenlin Yang
user
YZL0602
68118b63439785d7ff81a87d
Tong Liu
68118b63439785d7ff81a87e
Qingrong Xia
68118b63439785d7ff81a87f
Xinyu Duan
admin_assigned
2025-05-05T08:13:21.264Z
673ee647be09ba94d3971e6d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mRVXskb5zvNRtb6Unysa1.jpeg
xinyu duan
user
hetao06
68118b63439785d7ff81a880
Zhefeng Wang
68118b63439785d7ff81a881
Baoxing Huai
68118b63439785d7ff81a882
Min Zhang
Taming the Titans: A Survey of Efficient LLM Inference Serving
Large Language Models (LLMs) for Generative AI have achieved remarkable
progress, evolving into sophisticated and versatile tools widely adopted across
various domains and applications. However, the substantial memory overhead
caused by their vast number of parameters, combined with the high computational
demands of the attention mechanism, poses significant challenges in achieving
low latency and high throughput for LLM inference services. Recent
advancements, driven by groundbreaking research, have significantly accelerated
progress in this field. This paper provides a comprehensive survey of these
methods, covering fundamental instance-level approaches, in-depth cluster-level
strategies, emerging scenario directions, and other miscellaneous but important
areas. At the instance level, we review model placement, request scheduling,
decoding length prediction, storage management, and the disaggregation
paradigm. At the cluster level, we explore GPU cluster deployment,
multi-instance load balancing, and cloud service solutions. For emerging
scenarios, we organize the discussion around specific tasks, modules, and
auxiliary methods. To ensure a holistic overview, we also highlight several
niche yet critical areas. Finally, we outline potential research directions to
further advance the field of LLM inference serving.
A survey explores methods to enhance low latency and high throughput in Large Language Model inference by addressing memory overhead and computational demands of the attention mechanism.
Large Language Models Generative AI parameter overhead attention mechanism model placement request scheduling decoding length prediction storage management disaggregation GPU cluster deployment multi-instance load balancing cloud service solutions emerging scenarios research directions