title: Autellix: An Efficient Serving Engine for LLM Agents as General Programs
summary: Large language model (LLM) applications are evolving beyond simple chatbots
  into dynamic, general-purpose agentic programs, which scale LLM calls and
  output tokens to help AI agents reason, explore, and solve complex tasks.
  However, existing LLM serving systems ignore dependencies between programs and
  calls, missing significant opportunities for optimization. Our analysis reveals
  that programs submitted to LLM serving engines experience long cumulative wait
  times, primarily due to head-of-line blocking at both the individual LLM
  request and the program. To address this, we introduce Autellix, an LLM serving
  system that treats programs as first-class citizens to minimize their
  end-to-end latencies. Autellix intercepts LLM calls submitted by programs,
  enriching schedulers with program-level context. We propose two scheduling
  algorithms-for single-threaded and distributed programs-that preempt and
  prioritize LLM calls based on their programs' previously completed calls. Our
  evaluation demonstrates that across diverse LLMs and agentic workloads,
  Autellix improves throughput of programs by 4-15x at the same latency compared
  to state-of-the-art systems, such as vLLM.
paper_authors._id: 67b6a3fa09841367596a1db5
paper_authors.name: Michael Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:25:24.729Z
paper_authors._id: 654037be97949fd2304aab7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654037be97949fd2304aab7f/2cSME81gcwYa2OTeVlq5Q.jpeg
paper_authors.fullname: Michael Luo
paper_authors.type: user
paper_authors.user: michaelzhiluo
paper_authors._id: 67b6a3fa09841367596a1db6
paper_authors.name: Xiaoxiang Shi
paper_authors._id: 67b6a3fa09841367596a1db7
paper_authors.name: Colin Cai
paper_authors._id: 67b6a3fa09841367596a1db8
paper_authors.name: Tianjun Zhang
paper_authors._id: 67b6a3fa09841367596a1db9
paper_authors.name: Justin Wong
paper_authors._id: 67b6a3fa09841367596a1dba
paper_authors.name: Yichuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:35:50.487Z
paper_authors._id: 626e3449e7914f0d5ea78ad1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626e3449e7914f0d5ea78ad1/pVzdmdPMpNcxuj94qiIvB.jpeg
paper_authors.fullname: Yichuan
paper_authors.type: user
paper_authors.user: Chrisyichuan
paper_authors._id: 67b6a3fa09841367596a1dbb
paper_authors.name: Chi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:38:54.531Z
paper_authors._id: 67bea54186192d5a029f029b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mLRb5e_ucqlvPdexaVk7E.png
paper_authors.fullname: Chi Wang
paper_authors.type: user
paper_authors.user: chi-wang
paper_authors._id: 67b6a3fa09841367596a1dbc
paper_authors.name: Yanping Huang
paper_authors._id: 67b6a3fa09841367596a1dbd
paper_authors.name: Zhifeng Chen
paper_authors._id: 67b6a3fa09841367596a1dbe
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:10:25.466Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 67b6a3fa09841367596a1dbf
paper_authors.name: Ion Stoica
paper_title: Autellix: An Efficient Serving Engine for LLM Agents as General Programs
paper_summary: Large language model (LLM) applications are evolving beyond simple chatbots
  into dynamic, general-purpose agentic programs, which scale LLM calls and
  output tokens to help AI agents reason, explore, and solve complex tasks.
  However, existing LLM serving systems ignore dependencies between programs and
  calls, missing significant opportunities for optimization. Our analysis reveals
  that programs submitted to LLM serving engines experience long cumulative wait
  times, primarily due to head-of-line blocking at both the individual LLM
  request and the program. To address this, we introduce Autellix, an LLM serving
  system that treats programs as first-class citizens to minimize their
  end-to-end latencies. Autellix intercepts LLM calls submitted by programs,
  enriching schedulers with program-level context. We propose two scheduling
  algorithms-for single-threaded and distributed programs-that preempt and
  prioritize LLM calls based on their programs' previously completed calls. Our
  evaluation demonstrates that across diverse LLMs and agentic workloads,
  Autellix improves throughput of programs by 4-15x at the same latency compared
  to state-of-the-art systems, such as vLLM.
paper_ai_keywords: LLM (Large language model) agentic programs LLM serving systems head-of-line CV Autellix program-level context single-threaded programs distributed programs LLM calls vLLM