A Survey on Inference Engines for Large Language Models: Perspectives on
  Optimization and Efficiency
Large language models (LLMs) are widely applied in chatbots, code generators,
and search engines. Workloads such as chain-of-thought, complex reasoning, and
agent services significantly increase the inference cost by invoking the model
repeatedly. Optimization methods such as parallelism, compression, and caching
have been adopted to reduce costs, but the diverse service requirements make it
hard to select the right method. Recently, specialized LLM inference engines
have emerged as a key component for integrating the optimization methods into
service-oriented infrastructures. However, a systematic study on inference
engines is still lacking. This paper provides a comprehensive evaluation of 25
open-source and commercial inference engines. We examine each inference engine
in terms of ease-of-use, ease-of-deployment, general-purpose support,
scalability, and suitability for throughput- and latency-aware computation.
Furthermore, we explore the design goals of each inference engine by
investigating the optimization techniques it supports. In addition, we assess
the ecosystem maturity of open source inference engines and handle the
performance and cost policy of commercial solutions. We outline future research
directions that include support for complex LLM-based services, support of
various hardware, and enhanced security, offering practical guidance to
researchers and developers in selecting and designing optimized LLM inference
engines. We also provide a public repository to continually track developments
in this fast-evolving field:
https://github.com/sihyeong/Awesome-LLM-Inference-Engine
6819950bd55db085708dd2e5
Sihyeong Park
admin_assigned
2025-05-06T09:09:00.902Z
670cb786e73576f33a339144
/avatars/c172887c32878aebafd786061680ea1e.svg
Sihyeong Park
user
inputsh
6819950bd55db085708dd2e6
Sungryeol Jeon
6819950bd55db085708dd2e7
Chaelyn Lee
admin_assigned
2025-05-06T09:09:13.239Z
64aaa12a04e7b379fed24327
/avatars/327482e569c24ee4c97064f07ddd6de7.svg
Chaelyn Lee
user
oos2
6819950bd55db085708dd2e8
Seokhun Jeon
admin_assigned
2025-05-06T09:09:18.870Z
6719f17ac5837d514cfff13b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GnL4RCj7xncVhCIFN5y35.png
Seokhun Jeon
user
Devcow
6819950bd55db085708dd2e9
Byung-Soo Kim
6819950bd55db085708dd2ea
Jemin Lee
claimed_verified
2025-05-06T08:32:52.950Z
65b9dee19c4955ae7aee4954
/avatars/263f129605c7763185c49076174b891b.svg
Jemin Lee
user
leejaymin
A Survey on Inference Engines for Large Language Models: Perspectives on
  Optimization and Efficiency
Large language models (LLMs) are widely applied in chatbots, code generators,
and search engines. Workloads such as chain-of-thought, complex reasoning, and
agent services significantly increase the inference cost by invoking the model
repeatedly. Optimization methods such as parallelism, compression, and caching
have been adopted to reduce costs, but the diverse service requirements make it
hard to select the right method. Recently, specialized LLM inference engines
have emerged as a key component for integrating the optimization methods into
service-oriented infrastructures. However, a systematic study on inference
engines is still lacking. This paper provides a comprehensive evaluation of 25
open-source and commercial inference engines. We examine each inference engine
in terms of ease-of-use, ease-of-deployment, general-purpose support,
scalability, and suitability for throughput- and latency-aware computation.
Furthermore, we explore the design goals of each inference engine by
investigating the optimization techniques it supports. In addition, we assess
the ecosystem maturity of open source inference engines and handle the
performance and cost policy of commercial solutions. We outline future research
directions that include support for complex LLM-based services, support of
various hardware, and enhanced security, offering practical guidance to
researchers and developers in selecting and designing optimized LLM inference
engines. We also provide a public repository to continually track developments
in this fast-evolving field:
https://github.com/sihyeong/Awesome-LLM-Inference-Engine
A comprehensive evaluation of 25 open-source and commercial LLM inference engines across various criteria and optimization techniques is provided, with an outline of future research directions.
LLM large language models chain-of-thought complex reasoning agent services parallelism compression caching inference engines throughput-aware latency-aware optimization techniques ecosystem maturity performance cost policy security