title: Efficiently Serving LLM Reasoning Programs with Certaindex
summary: The rapid evolution of large language models (LLMs) has unlocked their
  capabilities in advanced reasoning tasks like mathematical problem-solving,
  code generation, and legal analysis. Central to this progress are
  inference-time reasoning algorithms, which refine outputs by exploring multiple
  solution paths, at the cost of increasing compute demands and response
  latencies. Existing serving systems fail to adapt to the scaling behaviors of
  these algorithms or the varying difficulty of queries, leading to inefficient
  resource use and unmet latency targets.
    We present Dynasor, a system that optimizes inference-time compute for LLM
  reasoning queries. Unlike traditional engines, Dynasor tracks and schedules
  requests within reasoning queries and uses Certaindex, a proxy that measures
  statistical reasoning progress based on model certainty, to guide compute
  allocation dynamically. Dynasor co-adapts scheduling with reasoning progress:
  it allocates more compute to hard queries, reduces compute for simpler ones,
  and terminates unpromising queries early, balancing accuracy, latency, and
  cost. On diverse datasets and algorithms, Dynasor reduces compute by up to 50%
  in batch processing and sustaining 3.3x higher query rates or 4.7x tighter
  latency SLOs in online serving.
paper_authors._id: 6773560ad86f2a718772598b
paper_authors.name: Yichao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:34.516Z
paper_authors._id: 62ba66296501b0ff15ba1075
paper_authors.avatarUrl: /avatars/de468727cb1240fd4b1f24a19fb237a6.svg
paper_authors.fullname: Yichao Fu
paper_authors.type: user
paper_authors.user: Viol2000
paper_authors._id: 6773560ad86f2a718772598c
paper_authors.name: Junda Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T08:36:19.944Z
paper_authors._id: 643839d9581e6bf0fa9c835e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643839d9581e6bf0fa9c835e/JxlgR-zQhms-rfF0sDxD8.jpeg
paper_authors.fullname: Junda Chen
paper_authors.type: user
paper_authors.user: GindaChen
paper_authors._id: 6773560ad86f2a718772598d
paper_authors.name: Siqi Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:32.277Z
paper_authors._id: 65621fd68631d43d2baf33b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZrNK7489lqq_vLeQgzhcR.png
paper_authors.fullname: siqi zhu
paper_authors.type: user
paper_authors.user: zsqzz
paper_authors._id: 6773560ad86f2a718772598e
paper_authors.name: Zheyu Fu
paper_authors._id: 6773560ad86f2a718772598f
paper_authors.name: Zhongdongming Dai
paper_authors._id: 6773560ad86f2a7187725990
paper_authors.name: Aurick Qiao
paper_authors._id: 6773560ad86f2a7187725991
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:13:29.981Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_title: Efficiently Serving LLM Reasoning Programs with Certaindex
paper_summary: The rapid evolution of large language models (LLMs) has unlocked their
  capabilities in advanced reasoning tasks like mathematical problem-solving,
  code generation, and legal analysis. Central to this progress are
  inference-time reasoning algorithms, which refine outputs by exploring multiple
  solution paths, at the cost of increasing compute demands and response
  latencies. Existing serving systems fail to adapt to the scaling behaviors of
  these algorithms or the varying difficulty of queries, leading to inefficient
  resource use and unmet latency targets.
    We present Dynasor, a system that optimizes inference-time compute for LLM
  reasoning queries. Unlike traditional engines, Dynasor tracks and schedules
  requests within reasoning queries and uses Certaindex, a proxy that measures
  statistical reasoning progress based on model certainty, to guide compute
  allocation dynamically. Dynasor co-adapts scheduling with reasoning progress:
  it allocates more compute to hard queries, reduces compute for simpler ones,
  and terminates unpromising queries early, balancing accuracy, latency, and
  cost. On diverse datasets and algorithms, Dynasor reduces compute by up to 50%
  in batch processing and sustaining 3.3x higher query rates or 4.7x tighter
  latency SLOs in online serving.
paper_ai_keywords: large language models (LLMs) advanced reasoning tasks mathematical problem-solving code generation legal analysis inference-time reasoning algorithms solution paths compute demands response latencies Dynasor Certaindex model certainty compute allocation batch processing query rates latency SLOs