title: A Survey on Inference Engines for Large Language Models: Perspectives on
    Optimization and Efficiency
summary: Large language models (LLMs) are widely applied in chatbots, code generators,
  and search engines. Workloads such as chain-of-thought, complex reasoning, and
  agent services significantly increase the inference cost by invoking the model
  repeatedly. Optimization methods such as parallelism, compression, and caching
  have been adopted to reduce costs, but the diverse service requirements make it
  hard to select the right method. Recently, specialized LLM inference engines
  have emerged as a key component for integrating the optimization methods into
  service-oriented infrastructures. However, a systematic study on inference
  engines is still lacking. This paper provides a comprehensive evaluation of 25
  open-source and commercial inference engines. We examine each inference engine
  in terms of ease-of-use, ease-of-deployment, general-purpose support,
  scalability, and suitability for throughput- and latency-aware computation.
  Furthermore, we explore the design goals of each inference engine by
  investigating the optimization techniques it supports. In addition, we assess
  the ecosystem maturity of open source inference engines and handle the
  performance and cost policy of commercial solutions. We outline future research
  directions that include support for complex LLM-based services, support of
  various hardware, and enhanced security, offering practical guidance to
  researchers and developers in selecting and designing optimized LLM inference
  engines. We also provide a public repository to continually track developments
  in this fast-evolving field:
  https://github.com/sihyeong/Awesome-LLM-Inference-Engine
paper_authors._id: 6819950bd55db085708dd2e5
paper_authors.name: Sihyeong Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:09:00.902Z
paper_authors._id: 670cb786e73576f33a339144
paper_authors.avatarUrl: /avatars/c172887c32878aebafd786061680ea1e.svg
paper_authors.fullname: Sihyeong Park
paper_authors.type: user
paper_authors.user: inputsh
paper_authors._id: 6819950bd55db085708dd2e6
paper_authors.name: Sungryeol Jeon
paper_authors._id: 6819950bd55db085708dd2e7
paper_authors.name: Chaelyn Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:09:13.239Z
paper_authors._id: 64aaa12a04e7b379fed24327
paper_authors.avatarUrl: /avatars/327482e569c24ee4c97064f07ddd6de7.svg
paper_authors.fullname: Chaelyn Lee
paper_authors.type: user
paper_authors.user: oos2
paper_authors._id: 6819950bd55db085708dd2e8
paper_authors.name: Seokhun Jeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:09:18.870Z
paper_authors._id: 6719f17ac5837d514cfff13b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GnL4RCj7xncVhCIFN5y35.png
paper_authors.fullname: Seokhun Jeon
paper_authors.type: user
paper_authors.user: Devcow
paper_authors._id: 6819950bd55db085708dd2e9
paper_authors.name: Byung-Soo Kim
paper_authors._id: 6819950bd55db085708dd2ea
paper_authors.name: Jemin Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:32:52.950Z
paper_authors._id: 65b9dee19c4955ae7aee4954
paper_authors.avatarUrl: /avatars/263f129605c7763185c49076174b891b.svg
paper_authors.fullname: Jemin Lee
paper_authors.type: user
paper_authors.user: leejaymin
paper_title: A Survey on Inference Engines for Large Language Models: Perspectives on
    Optimization and Efficiency
paper_summary: Large language models (LLMs) are widely applied in chatbots, code generators,
  and search engines. Workloads such as chain-of-thought, complex reasoning, and
  agent services significantly increase the inference cost by invoking the model
  repeatedly. Optimization methods such as parallelism, compression, and caching
  have been adopted to reduce costs, but the diverse service requirements make it
  hard to select the right method. Recently, specialized LLM inference engines
  have emerged as a key component for integrating the optimization methods into
  service-oriented infrastructures. However, a systematic study on inference
  engines is still lacking. This paper provides a comprehensive evaluation of 25
  open-source and commercial inference engines. We examine each inference engine
  in terms of ease-of-use, ease-of-deployment, general-purpose support,
  scalability, and suitability for throughput- and latency-aware computation.
  Furthermore, we explore the design goals of each inference engine by
  investigating the optimization techniques it supports. In addition, we assess
  the ecosystem maturity of open source inference engines and handle the
  performance and cost policy of commercial solutions. We outline future research
  directions that include support for complex LLM-based services, support of
  various hardware, and enhanced security, offering practical guidance to
  researchers and developers in selecting and designing optimized LLM inference
  engines. We also provide a public repository to continually track developments
  in this fast-evolving field:
  https://github.com/sihyeong/Awesome-LLM-Inference-Engine
paper_ai_summary: A comprehensive evaluation of 25 open-source and commercial LLM inference engines across various criteria and optimization techniques is provided, with an outline of future research directions.
paper_ai_keywords: LLM large language models chain-of-thought complex reasoning agent services parallelism compression caching inference engines throughput-aware latency-aware optimization techniques ecosystem maturity performance cost policy security