From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and
  BenchBuilder Pipeline
The rapid evolution of language models has necessitated the development of
more challenging benchmarks. Current static benchmarks often struggle to
consistently distinguish between the capabilities of different models and fail
to align with real-world user preferences. On the other hand, live
crowd-sourced platforms like the Chatbot Arena collect a wide range of natural
prompts and user feedback. However, these prompts vary in sophistication and
the feedback cannot be applied offline to new models. In order to ensure that
benchmarks keep up with the pace of LLM development, we address how one can
evaluate benchmarks on their ability to confidently separate models and their
alignment with human preference. Under these principles, we developed
BenchBuilder, a living benchmark that filters high-quality prompts from live
data sources to enable offline evaluation on fresh, challenging prompts.
BenchBuilder identifies seven indicators of a high-quality prompt, such as the
requirement for domain knowledge, and utilizes an LLM annotator to select a
high-quality subset of prompts from various topic clusters. The LLM evaluation
process employs an LLM judge to ensure a fully automated, high-quality, and
constantly updating benchmark. We apply BenchBuilder on prompts from the
Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from
a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence
intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with
human preference rankings, all at a cost of only $25 and without human
labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides
a valuable tool for developers, enabling them to extract high-quality
benchmarks from extensive data with minimal effort.
667304d105a87a94e686c084
Tianle Li
extracted_confirmed
2024-06-19T16:28:32.308Z
64ab7f188a1a9187c219e597
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab7f188a1a9187c219e597/cieU4bdmPIn9dTNIVf3n7.jpeg
Tianle Li
user
Timmli
667304d105a87a94e686c085
Wei-Lin Chiang
extracted_confirmed
2024-06-19T16:21:23.012Z
63071772fd79b417f1b92291
/avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
Wei-Lin Chiang
user
weichiang
667304d105a87a94e686c086
Evan Frick
admin_assigned
2024-06-20T07:42:23.593Z
64e8ff509d939716d00f3690
/avatars/028713c83e99166fa40bd54980bc12ec.svg
Evan Frick
user
evanfrick
667304d105a87a94e686c087
Lisa Dunlap
admin_assigned
2024-06-20T07:42:29.194Z
63f7ccb8ad35fabd915171a8
/avatars/786bc609e6305730f0f2025ae5c27f82.svg
Lisa Dunlap
user
lisabdunlap
667304d105a87a94e686c088
Tianhao Wu
667304d105a87a94e686c089
Banghua Zhu
admin_assigned
2024-06-20T07:43:00.682Z
647b8885aba7062fe5c32000
/avatars/184e6199e20ad63b3de12436bdd00a00.svg
Banghua Zhu
user
banghua
667304d105a87a94e686c08a
Joseph E. Gonzalez
admin_assigned
2024-06-20T07:43:12.033Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
667304d105a87a94e686c08b
Ion Stoica
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and
  BenchBuilder Pipeline
The rapid evolution of language models has necessitated the development of
more challenging benchmarks. Current static benchmarks often struggle to
consistently distinguish between the capabilities of different models and fail
to align with real-world user preferences. On the other hand, live
crowd-sourced platforms like the Chatbot Arena collect a wide range of natural
prompts and user feedback. However, these prompts vary in sophistication and
the feedback cannot be applied offline to new models. In order to ensure that
benchmarks keep up with the pace of LLM development, we address how one can
evaluate benchmarks on their ability to confidently separate models and their
alignment with human preference. Under these principles, we developed
BenchBuilder, a living benchmark that filters high-quality prompts from live
data sources to enable offline evaluation on fresh, challenging prompts.
BenchBuilder identifies seven indicators of a high-quality prompt, such as the
requirement for domain knowledge, and utilizes an LLM annotator to select a
high-quality subset of prompts from various topic clusters. The LLM evaluation
process employs an LLM judge to ensure a fully automated, high-quality, and
constantly updating benchmark. We apply BenchBuilder on prompts from the
Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from
a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence
intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with
human preference rankings, all at a cost of only $25 and without human
labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides
a valuable tool for developers, enabling them to extract high-quality
benchmarks from extensive data with minimal effort.
BenchBuilder enhances language model evaluation by creating a living benchmark that automates the selection and evaluation of high-quality prompts using LLM annotators and judges, ensuring continuous alignment with human preferences.
language models BenchBuilder Chatbot Arena high-quality prompts domain knowledge LLM annotator LLM judge Arena-Hard-Auto confidence intervals human preference rankings