title: Efficient LLM Scheduling by Learning to Rank
summary: In Large Language Model (LLM) inference, the output length of an LLM request
  is typically regarded as not known a priori. Consequently, most LLM serving
  systems employ a simple First-come-first-serve (FCFS) scheduling strategy,
  leading to Head-Of-Line (HOL) blocking and reduced throughput and service
  quality. In this paper, we reexamine this assumption -- we show that, although
  predicting the exact generation length of each request is infeasible, it is
  possible to predict the relative ranks of output lengths in a batch of
  requests, using learning to rank. The ranking information offers valuable
  guidance for scheduling requests. Building on this insight, we develop a novel
  scheduler for LLM inference and serving that can approximate the
  shortest-job-first (SJF) schedule better than existing approaches. We integrate
  this scheduler with the state-of-the-art LLM serving system and show
  significant performance improvement in several important applications: 2.8x
  lower latency in chatbot serving and 6.5x higher throughput in synthetic data
  generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
paper_authors._id: 66d03dcc62252ca2b74f16d7
paper_authors.name: Yichao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T09:25:30.036Z
paper_authors._id: 62ba66296501b0ff15ba1075
paper_authors.avatarUrl: /avatars/de468727cb1240fd4b1f24a19fb237a6.svg
paper_authors.fullname: Yichao Fu
paper_authors.type: user
paper_authors.user: Viol2000
paper_authors._id: 66d03dcc62252ca2b74f16d8
paper_authors.name: Siqi Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T15:22:11.111Z
paper_authors._id: 65621fd68631d43d2baf33b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZrNK7489lqq_vLeQgzhcR.png
paper_authors.fullname: siqi zhu
paper_authors.type: user
paper_authors.user: zsqzz
paper_authors._id: 66d03dcc62252ca2b74f16d9
paper_authors.name: Runlong Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:04:36.449Z
paper_authors._id: 65d7ed4823e83e1591beacc7
paper_authors.avatarUrl: /avatars/2a6714a2a7bbd591f6b726a7330bafbc.svg
paper_authors.fullname: Su
paper_authors.type: user
paper_authors.user: r3su9
paper_authors._id: 66d03dcc62252ca2b74f16da
paper_authors.name: Aurick Qiao
paper_authors._id: 66d03dcc62252ca2b74f16db
paper_authors.name: Ion Stoica
paper_authors._id: 66d03dcc62252ca2b74f16dc
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:14:28.234Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_title: Efficient LLM Scheduling by Learning to Rank
paper_summary: In Large Language Model (LLM) inference, the output length of an LLM request
  is typically regarded as not known a priori. Consequently, most LLM serving
  systems employ a simple First-come-first-serve (FCFS) scheduling strategy,
  leading to Head-Of-Line (HOL) blocking and reduced throughput and service
  quality. In this paper, we reexamine this assumption -- we show that, although
  predicting the exact generation length of each request is infeasible, it is
  possible to predict the relative ranks of output lengths in a batch of
  requests, using learning to rank. The ranking information offers valuable
  guidance for scheduling requests. Building on this insight, we develop a novel
  scheduler for LLM inference and serving that can approximate the
  shortest-job-first (SJF) schedule better than existing approaches. We integrate
  this scheduler with the state-of-the-art LLM serving system and show
  significant performance improvement in several important applications: 2.8x
  lower latency in chatbot serving and 6.5x higher throughput in synthetic data
  generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
paper_ai_keywords: learning to rank shortest-job-first (SJF) schedule chatbot serving synthetic data generation