Autellix: An Efficient Serving Engine for LLM Agents as General Programs
Large language model (LLM) applications are evolving beyond simple chatbots
into dynamic, general-purpose agentic programs, which scale LLM calls and
output tokens to help AI agents reason, explore, and solve complex tasks.
However, existing LLM serving systems ignore dependencies between programs and
calls, missing significant opportunities for optimization. Our analysis reveals
that programs submitted to LLM serving engines experience long cumulative wait
times, primarily due to head-of-line blocking at both the individual LLM
request and the program. To address this, we introduce Autellix, an LLM serving
system that treats programs as first-class citizens to minimize their
end-to-end latencies. Autellix intercepts LLM calls submitted by programs,
enriching schedulers with program-level context. We propose two scheduling
algorithms-for single-threaded and distributed programs-that preempt and
prioritize LLM calls based on their programs' previously completed calls. Our
evaluation demonstrates that across diverse LLMs and agentic workloads,
Autellix improves throughput of programs by 4-15x at the same latency compared
to state-of-the-art systems, such as vLLM.
67b6a3fa09841367596a1db5
Michael Luo
claimed_verified
2025-02-24T09:25:24.729Z
654037be97949fd2304aab7f
https://cdn-avatars.huggingface.co/v1/production/uploads/654037be97949fd2304aab7f/2cSME81gcwYa2OTeVlq5Q.jpeg
Michael Luo
user
michaelzhiluo
67b6a3fa09841367596a1db6
Xiaoxiang Shi
67b6a3fa09841367596a1db7
Colin Cai
67b6a3fa09841367596a1db8
Tianjun Zhang
67b6a3fa09841367596a1db9
Justin Wong
67b6a3fa09841367596a1dba
Yichuan Wang
claimed_verified
2025-02-20T09:35:50.487Z
626e3449e7914f0d5ea78ad1
https://cdn-avatars.huggingface.co/v1/production/uploads/626e3449e7914f0d5ea78ad1/pVzdmdPMpNcxuj94qiIvB.jpeg
Yichuan
user
Chrisyichuan
67b6a3fa09841367596a1dbb
Chi Wang
claimed_verified
2025-02-26T08:38:54.531Z
67bea54186192d5a029f029b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mLRb5e_ucqlvPdexaVk7E.png
Chi Wang
user
chi-wang
67b6a3fa09841367596a1dbc
Yanping Huang
67b6a3fa09841367596a1dbd
Zhifeng Chen
67b6a3fa09841367596a1dbe
Joseph E. Gonzalez
admin_assigned
2025-02-20T16:10:25.466Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
67b6a3fa09841367596a1dbf
Ion Stoica
Autellix: An Efficient Serving Engine for LLM Agents as General Programs
Large language model (LLM) applications are evolving beyond simple chatbots
into dynamic, general-purpose agentic programs, which scale LLM calls and
output tokens to help AI agents reason, explore, and solve complex tasks.
However, existing LLM serving systems ignore dependencies between programs and
calls, missing significant opportunities for optimization. Our analysis reveals
that programs submitted to LLM serving engines experience long cumulative wait
times, primarily due to head-of-line blocking at both the individual LLM
request and the program. To address this, we introduce Autellix, an LLM serving
system that treats programs as first-class citizens to minimize their
end-to-end latencies. Autellix intercepts LLM calls submitted by programs,
enriching schedulers with program-level context. We propose two scheduling
algorithms-for single-threaded and distributed programs-that preempt and
prioritize LLM calls based on their programs' previously completed calls. Our
evaluation demonstrates that across diverse LLMs and agentic workloads,
Autellix improves throughput of programs by 4-15x at the same latency compared
to state-of-the-art systems, such as vLLM.
Autellix improves LLM serving system efficiency by prioritizing program-level dependencies, significantly boosting throughput while maintaining latency.
LLM large language model agentic programs cumulative wait times head-of-line blocking scheduling algorithms preemptive scheduling program-level context