title: From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and
    BenchBuilder Pipeline
summary: The rapid evolution of language models has necessitated the development of
  more challenging benchmarks. Current static benchmarks often struggle to
  consistently distinguish between the capabilities of different models and fail
  to align with real-world user preferences. On the other hand, live
  crowd-sourced platforms like the Chatbot Arena collect a wide range of natural
  prompts and user feedback. However, these prompts vary in sophistication and
  the feedback cannot be applied offline to new models. In order to ensure that
  benchmarks keep up with the pace of LLM development, we address how one can
  evaluate benchmarks on their ability to confidently separate models and their
  alignment with human preference. Under these principles, we developed
  BenchBuilder, a living benchmark that filters high-quality prompts from live
  data sources to enable offline evaluation on fresh, challenging prompts.
  BenchBuilder identifies seven indicators of a high-quality prompt, such as the
  requirement for domain knowledge, and utilizes an LLM annotator to select a
  high-quality subset of prompts from various topic clusters. The LLM evaluation
  process employs an LLM judge to ensure a fully automated, high-quality, and
  constantly updating benchmark. We apply BenchBuilder on prompts from the
  Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from
  a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence
  intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with
  human preference rankings, all at a cost of only $25 and without human
  labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides
  a valuable tool for developers, enabling them to extract high-quality
  benchmarks from extensive data with minimal effort.
paper_authors._id: 667304d105a87a94e686c084
paper_authors.name: Tianle Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-19T16:28:32.308Z
paper_authors._id: 64ab7f188a1a9187c219e597
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab7f188a1a9187c219e597/cieU4bdmPIn9dTNIVf3n7.jpeg
paper_authors.fullname: Tianle Li
paper_authors.type: user
paper_authors.user: Timmli
paper_authors._id: 667304d105a87a94e686c085
paper_authors.name: Wei-Lin Chiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-19T16:21:23.012Z
paper_authors._id: 63071772fd79b417f1b92291
paper_authors.avatarUrl: /avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
paper_authors.fullname: Wei-Lin Chiang
paper_authors.type: user
paper_authors.user: weichiang
paper_authors._id: 667304d105a87a94e686c086
paper_authors.name: Evan Frick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:42:23.593Z
paper_authors._id: 64e8ff509d939716d00f3690
paper_authors.avatarUrl: /avatars/028713c83e99166fa40bd54980bc12ec.svg
paper_authors.fullname: Evan Frick
paper_authors.type: user
paper_authors.user: evanfrick
paper_authors._id: 667304d105a87a94e686c087
paper_authors.name: Lisa Dunlap
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:42:29.194Z
paper_authors._id: 63f7ccb8ad35fabd915171a8
paper_authors.avatarUrl: /avatars/786bc609e6305730f0f2025ae5c27f82.svg
paper_authors.fullname: Lisa Dunlap
paper_authors.type: user
paper_authors.user: lisabdunlap
paper_authors._id: 667304d105a87a94e686c088
paper_authors.name: Tianhao Wu
paper_authors._id: 667304d105a87a94e686c089
paper_authors.name: Banghua Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:43:00.682Z
paper_authors._id: 647b8885aba7062fe5c32000
paper_authors.avatarUrl: /avatars/128ceae78490110ae41202851e84d58e.svg
paper_authors.fullname: Banghua Zhu
paper_authors.type: user
paper_authors.user: banghua
paper_authors._id: 667304d105a87a94e686c08a
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:43:12.033Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 667304d105a87a94e686c08b
paper_authors.name: Ion Stoica
paper_title: From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and
    BenchBuilder Pipeline
paper_summary: The rapid evolution of language models has necessitated the development of
  more challenging benchmarks. Current static benchmarks often struggle to
  consistently distinguish between the capabilities of different models and fail
  to align with real-world user preferences. On the other hand, live
  crowd-sourced platforms like the Chatbot Arena collect a wide range of natural
  prompts and user feedback. However, these prompts vary in sophistication and
  the feedback cannot be applied offline to new models. In order to ensure that
  benchmarks keep up with the pace of LLM development, we address how one can
  evaluate benchmarks on their ability to confidently separate models and their
  alignment with human preference. Under these principles, we developed
  BenchBuilder, a living benchmark that filters high-quality prompts from live
  data sources to enable offline evaluation on fresh, challenging prompts.
  BenchBuilder identifies seven indicators of a high-quality prompt, such as the
  requirement for domain knowledge, and utilizes an LLM annotator to select a
  high-quality subset of prompts from various topic clusters. The LLM evaluation
  process employs an LLM judge to ensure a fully automated, high-quality, and
  constantly updating benchmark. We apply BenchBuilder on prompts from the
  Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from
  a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence
  intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with
  human preference rankings, all at a cost of only $25 and without human
  labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides
  a valuable tool for developers, enabling them to extract high-quality
  benchmarks from extensive data with minimal effort.
paper_ai_summary: BenchBuilder enhances language model evaluation by creating a living benchmark that automates the selection and evaluation of high-quality prompts using LLM annotators and judges, ensuring continuous alignment with human preferences.
paper_ai_keywords: language models BenchBuilder Chatbot Arena high-quality prompts domain knowledge LLM annotator LLM judge Arena-Hard-Auto confidence intervals human preference rankings