Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Large Language Models (LLMs) have unlocked new capabilities and applications;
however, evaluating the alignment with human preferences still poses
significant challenges. To address this issue, we introduce Chatbot Arena, an
open platform for evaluating LLMs based on human preferences. Our methodology
employs a pairwise comparison approach and leverages input from a diverse user
base through crowdsourcing. The platform has been operational for several
months, amassing over 240K votes. This paper describes the platform, analyzes
the data we have collected so far, and explains the tried-and-true statistical
methods we are using for efficient and accurate evaluation and ranking of
models. We confirm that the crowdsourced questions are sufficiently diverse and
discriminating and that the crowdsourced human votes are in good agreement with
those of expert raters. These analyses collectively establish a robust
foundation for the credibility of Chatbot Arena. Because of its unique value
and openness, Chatbot Arena has emerged as one of the most referenced LLM
leaderboards, widely cited by leading LLM developers and companies. Our demo is
publicly available at https://chat.lmsys.org.
65ea7f05e36ad838b84b8001
Wei-Lin Chiang
extracted_confirmed
2024-03-08T03:03:34.163Z
63071772fd79b417f1b92291
/avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
Wei-Lin Chiang
user
weichiang
65ea7f05e36ad838b84b8002
Lianmin Zheng
65ea7f05e36ad838b84b8003
Ying Sheng
65ea7f05e36ad838b84b8004
Anastasios Nikolas Angelopoulos
admin_assigned
2024-03-08T09:06:04.862Z
62fc26dde44837de54496319
https://cdn-avatars.huggingface.co/v1/production/uploads/1660692172781-noauth.png
Anastasios Nikolas Angelopoulos
user
angelopoulos
65ea7f05e36ad838b84b8005
Tianle Li
admin_assigned
2024-03-08T09:06:18.846Z
64ab7f188a1a9187c219e597
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab7f188a1a9187c219e597/cieU4bdmPIn9dTNIVf3n7.jpeg
Tianle Li
user
Timmli
65ea7f05e36ad838b84b8006
Dacheng Li
admin_assigned
2024-03-08T09:06:25.806Z
63715b25ffc0489ed7d1f415
https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
Dacheng Li
user
DachengLi
65ea7f05e36ad838b84b8007
Hao Zhang
claimed_verified
2025-01-01T20:14:57.714Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
65ea7f05e36ad838b84b8008
Banghua Zhu
admin_assigned
2024-03-08T09:06:52.744Z
647b8885aba7062fe5c32000
/avatars/184e6199e20ad63b3de12436bdd00a00.svg
Banghua Zhu
user
banghua
65ea7f05e36ad838b84b8009
Michael Jordan
65ea7f05e36ad838b84b800a
Joseph E. Gonzalez
admin_assigned
2024-03-08T09:07:21.899Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
65ea7f05e36ad838b84b800b
Ion Stoica
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Large Language Models (LLMs) have unlocked new capabilities and applications;
however, evaluating the alignment with human preferences still poses
significant challenges. To address this issue, we introduce Chatbot Arena, an
open platform for evaluating LLMs based on human preferences. Our methodology
employs a pairwise comparison approach and leverages input from a diverse user
base through crowdsourcing. The platform has been operational for several
months, amassing over 240K votes. This paper describes the platform, analyzes
the data we have collected so far, and explains the tried-and-true statistical
methods we are using for efficient and accurate evaluation and ranking of
models. We confirm that the crowdsourced questions are sufficiently diverse and
discriminating and that the crowdsourced human votes are in good agreement with
those of expert raters. These analyses collectively establish a robust
foundation for the credibility of Chatbot Arena. Because of its unique value
and openness, Chatbot Arena has emerged as one of the most referenced LLM
leaderboards, widely cited by leading LLM developers and companies. Our demo is
publicly available at https://chat.lmsys.org.
Chatbot Arena, an open platform, evaluates LLMs based on human preferences using pairwise comparisons and crowdsourced votes, confirming its credibility through diverse and accurate data analysis.
Large Language Models Chatbot Arena pairwise comparison crowdsourcing statistical methods LLM leaderboards