title: Prompt-to-Leaderboard
summary: Large language model (LLM) evaluations typically rely on aggregated metrics
  like accuracy or human preference, averaging across users and prompts. This
  averaging obscures user- and prompt-specific variations in model performance.
  To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces
  leaderboards specific to a prompt. The core idea is to train an LLM taking
  natural language prompts as input to output a vector of Bradley-Terry
  coefficients which are then used to predict the human preference vote. The
  resulting prompt-dependent leaderboards allow for unsupervised task-specific
  evaluation, optimal routing of queries to models, personalization, and
  automated evaluation of model strengths and weaknesses. Data from Chatbot Arena
  suggest that P2L better captures the nuanced landscape of language model
  performance than the averaged leaderboard. Furthermore, our findings suggest
  that P2L's ability to produce prompt-specific evaluations follows a power law
  scaling similar to that observed in LLMs themselves. In January 2025, the
  router we trained based on this methodology achieved the \#1 spot in the
  Chatbot Arena leaderboard. Our code is available at this GitHub link:
  https://github.com/lmarena/p2l.
paper_authors._id: 67b8e77477a3ed169f302415
paper_authors.name: Evan Frick
paper_authors._id: 67b8e77477a3ed169f302416
paper_authors.name: Connor Chen
paper_authors._id: 67b8e77477a3ed169f302417
paper_authors.name: Joseph Tennyson
paper_authors._id: 67b8e77477a3ed169f302418
paper_authors.name: Tianle Li
paper_authors._id: 67b8e77477a3ed169f302419
paper_authors.name: Wei-Lin Chiang
paper_authors._id: 67b8e77477a3ed169f30241a
paper_authors.name: Anastasios N. Angelopoulos
paper_authors._id: 67b8e77477a3ed169f30241b
paper_authors.name: Ion Stoica
paper_title: Prompt-to-Leaderboard
paper_summary: Large language model (LLM) evaluations typically rely on aggregated metrics
  like accuracy or human preference, averaging across users and prompts. This
  averaging obscures user- and prompt-specific variations in model performance.
  To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces
  leaderboards specific to a prompt. The core idea is to train an LLM taking
  natural language prompts as input to output a vector of Bradley-Terry
  coefficients which are then used to predict the human preference vote. The
  resulting prompt-dependent leaderboards allow for unsupervised task-specific
  evaluation, optimal routing of queries to models, personalization, and
  automated evaluation of model strengths and weaknesses. Data from Chatbot Arena
  suggest that P2L better captures the nuanced landscape of language model
  performance than the averaged leaderboard. Furthermore, our findings suggest
  that P2L's ability to produce prompt-specific evaluations follows a power law
  scaling similar to that observed in LLMs themselves. In January 2025, the
  router we trained based on this methodology achieved the \#1 spot in the
  Chatbot Arena leaderboard. Our code is available at this GitHub link:
  https://github.com/lmarena/p2l.
paper_ai_keywords: Prompt-to-Leaderboard (P2L) Large-scale model (LSM) Bradley-Terry coefficients Chatbot Arena power law scaling