title: Taming the Titans: A Survey of Efficient LLM Inference Serving
summary: Large Language Models (LLMs) for Generative AI have achieved remarkable
  progress, evolving into sophisticated and versatile tools widely adopted across
  various domains and applications. However, the substantial memory overhead
  caused by their vast number of parameters, combined with the high computational
  demands of the attention mechanism, poses significant challenges in achieving
  low latency and high throughput for LLM inference services. Recent
  advancements, driven by groundbreaking research, have significantly accelerated
  progress in this field. This paper provides a comprehensive survey of these
  methods, covering fundamental instance-level approaches, in-depth cluster-level
  strategies, emerging scenario directions, and other miscellaneous but important
  areas. At the instance level, we review model placement, request scheduling,
  decoding length prediction, storage management, and the disaggregation
  paradigm. At the cluster level, we explore GPU cluster deployment,
  multi-instance load balancing, and cloud service solutions. For emerging
  scenarios, we organize the discussion around specific tasks, modules, and
  auxiliary methods. To ensure a holistic overview, we also highlight several
  niche yet critical areas. Finally, we outline potential research directions to
  further advance the field of LLM inference serving.
paper_authors._id: 68118b63439785d7ff81a879
paper_authors.name: Ranran Zhen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:15.143Z
paper_authors._id: 647e96507f9ad5e44bac50bd
paper_authors.avatarUrl: /avatars/2551f99401540676cfe5cd0ed99e70c1.svg
paper_authors.fullname: Ranran Zhen
paper_authors.type: user
paper_authors.user: zenRRan
paper_authors._id: 68118b63439785d7ff81a87a
paper_authors.name: Juntao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T08:12:36.237Z
paper_authors._id: 6670e285b0c03c4e9d6e0985
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uCZHm4gKSHZ2b0hpHWgZv.jpeg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: douvleplus
paper_authors._id: 68118b63439785d7ff81a87b
paper_authors.name: Yixin Ji
paper_authors._id: 68118b63439785d7ff81a87c
paper_authors.name: Zhenlin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T08:13:47.806Z
paper_authors._id: 64023a3dcad36eb2b0f9c2c0
paper_authors.avatarUrl: /avatars/306ce26d991ec58da89ccec37922d906.svg
paper_authors.fullname: Zhenlin Yang
paper_authors.type: user
paper_authors.user: YZL0602
paper_authors._id: 68118b63439785d7ff81a87d
paper_authors.name: Tong Liu
paper_authors._id: 68118b63439785d7ff81a87e
paper_authors.name: Qingrong Xia
paper_authors._id: 68118b63439785d7ff81a87f
paper_authors.name: Xinyu Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T08:13:21.264Z
paper_authors._id: 673ee647be09ba94d3971e6d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mRVXskb5zvNRtb6Unysa1.jpeg
paper_authors.fullname: xinyu duan
paper_authors.type: user
paper_authors.user: hetao06
paper_authors._id: 68118b63439785d7ff81a880
paper_authors.name: Zhefeng Wang
paper_authors._id: 68118b63439785d7ff81a881
paper_authors.name: Baoxing Huai
paper_authors._id: 68118b63439785d7ff81a882
paper_authors.name: Min Zhang
paper_title: Taming the Titans: A Survey of Efficient LLM Inference Serving
paper_summary: Large Language Models (LLMs) for Generative AI have achieved remarkable
  progress, evolving into sophisticated and versatile tools widely adopted across
  various domains and applications. However, the substantial memory overhead
  caused by their vast number of parameters, combined with the high computational
  demands of the attention mechanism, poses significant challenges in achieving
  low latency and high throughput for LLM inference services. Recent
  advancements, driven by groundbreaking research, have significantly accelerated
  progress in this field. This paper provides a comprehensive survey of these
  methods, covering fundamental instance-level approaches, in-depth cluster-level
  strategies, emerging scenario directions, and other miscellaneous but important
  areas. At the instance level, we review model placement, request scheduling,
  decoding length prediction, storage management, and the disaggregation
  paradigm. At the cluster level, we explore GPU cluster deployment,
  multi-instance load balancing, and cloud service solutions. For emerging
  scenarios, we organize the discussion around specific tasks, modules, and
  auxiliary methods. To ensure a holistic overview, we also highlight several
  niche yet critical areas. Finally, we outline potential research directions to
  further advance the field of LLM inference serving.
paper_ai_summary: A survey explores methods to enhance low latency and high throughput in Large Language Model inference by addressing memory overhead and computational demands of the attention mechanism.
paper_ai_keywords: Large Language Models Generative AI parameter overhead attention mechanism model placement request scheduling decoding length prediction storage management disaggregation GPU cluster deployment multi-instance load balancing cloud service solutions emerging scenarios research directions