Efficient LLM Scheduling by Learning to Rank
In Large Language Model (LLM) inference, the output length of an LLM request
is typically regarded as not known a priori. Consequently, most LLM serving
systems employ a simple First-come-first-serve (FCFS) scheduling strategy,
leading to Head-Of-Line (HOL) blocking and reduced throughput and service
quality. In this paper, we reexamine this assumption -- we show that, although
predicting the exact generation length of each request is infeasible, it is
possible to predict the relative ranks of output lengths in a batch of
requests, using learning to rank. The ranking information offers valuable
guidance for scheduling requests. Building on this insight, we develop a novel
scheduler for LLM inference and serving that can approximate the
shortest-job-first (SJF) schedule better than existing approaches. We integrate
this scheduler with the state-of-the-art LLM serving system and show
significant performance improvement in several important applications: 2.8x
lower latency in chatbot serving and 6.5x higher throughput in synthetic data
generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
66d03dcc62252ca2b74f16d7
Yichao Fu
claimed_verified
2024-08-29T09:25:30.036Z
62ba66296501b0ff15ba1075
/avatars/de468727cb1240fd4b1f24a19fb237a6.svg
Yichao Fu
user
Viol2000
66d03dcc62252ca2b74f16d8
Siqi Zhu
claimed_verified
2024-08-29T15:22:11.111Z
65621fd68631d43d2baf33b2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1JemNCnPkS1mE3SNygsE2.png
siqi zhu
user
zsqzz
66d03dcc62252ca2b74f16d9
Runlong Su
claimed_verified
2025-02-11T10:04:36.449Z
65d7ed4823e83e1591beacc7
/avatars/2a6714a2a7bbd591f6b726a7330bafbc.svg
Su
user
r3su9
66d03dcc62252ca2b74f16da
Aurick Qiao
66d03dcc62252ca2b74f16db
Ion Stoica
66d03dcc62252ca2b74f16dc
Hao Zhang
claimed_verified
2025-01-01T20:14:28.234Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
Efficient LLM Scheduling by Learning to Rank
In Large Language Model (LLM) inference, the output length of an LLM request
is typically regarded as not known a priori. Consequently, most LLM serving
systems employ a simple First-come-first-serve (FCFS) scheduling strategy,
leading to Head-Of-Line (HOL) blocking and reduced throughput and service
quality. In this paper, we reexamine this assumption -- we show that, although
predicting the exact generation length of each request is infeasible, it is
possible to predict the relative ranks of output lengths in a batch of
requests, using learning to rank. The ranking information offers valuable
guidance for scheduling requests. Building on this insight, we develop a novel
scheduler for LLM inference and serving that can approximate the
shortest-job-first (SJF) schedule better than existing approaches. We integrate
this scheduler with the state-of-the-art LLM serving system and show
significant performance improvement in several important applications: 2.8x
lower latency in chatbot serving and 6.5x higher throughput in synthetic data
generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git
A novel scheduler using learning to rank improves LLM inference performance by better approximating shortest-job-first scheduling.
Large Language Model LLM First-come-first-serve FCFS Head-Of-Line blocking HOL blocking learning to rank shortest-job-first SJF chatbot serving synthetic data generation