title: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
summary: Large Language Models (LLMs) have unlocked new capabilities and applications;
  however, evaluating the alignment with human preferences still poses
  significant challenges. To address this issue, we introduce Chatbot Arena, an
  open platform for evaluating LLMs based on human preferences. Our methodology
  employs a pairwise comparison approach and leverages input from a diverse user
  base through crowdsourcing. The platform has been operational for several
  months, amassing over 240K votes. This paper describes the platform, analyzes
  the data we have collected so far, and explains the tried-and-true statistical
  methods we are using for efficient and accurate evaluation and ranking of
  models. We confirm that the crowdsourced questions are sufficiently diverse and
  discriminating and that the crowdsourced human votes are in good agreement with
  those of expert raters. These analyses collectively establish a robust
  foundation for the credibility of Chatbot Arena. Because of its unique value
  and openness, Chatbot Arena has emerged as one of the most referenced LLM
  leaderboards, widely cited by leading LLM developers and companies. Our demo is
  publicly available at https://chat.lmsys.org.
paper_authors._id: 65ea7f05e36ad838b84b8001
paper_authors.name: Wei-Lin Chiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-08T03:03:34.163Z
paper_authors._id: 63071772fd79b417f1b92291
paper_authors.avatarUrl: /avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
paper_authors.fullname: Wei-Lin Chiang
paper_authors.type: user
paper_authors.user: weichiang
paper_authors._id: 65ea7f05e36ad838b84b8002
paper_authors.name: Lianmin Zheng
paper_authors._id: 65ea7f05e36ad838b84b8003
paper_authors.name: Ying Sheng
paper_authors._id: 65ea7f05e36ad838b84b8004
paper_authors.name: Anastasios Nikolas Angelopoulos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:06:04.862Z
paper_authors._id: 62fc26dde44837de54496319
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660692172781-noauth.png
paper_authors.fullname: Anastasios Nikolas Angelopoulos
paper_authors.type: user
paper_authors.user: angelopoulos
paper_authors._id: 65ea7f05e36ad838b84b8005
paper_authors.name: Tianle Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:06:18.846Z
paper_authors._id: 64ab7f188a1a9187c219e597
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab7f188a1a9187c219e597/cieU4bdmPIn9dTNIVf3n7.jpeg
paper_authors.fullname: Tianle Li
paper_authors.type: user
paper_authors.user: Timmli
paper_authors._id: 65ea7f05e36ad838b84b8006
paper_authors.name: Dacheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:06:25.806Z
paper_authors._id: 63715b25ffc0489ed7d1f415
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
paper_authors.fullname: Dacheng Li
paper_authors.type: user
paper_authors.user: DachengLi
paper_authors._id: 65ea7f05e36ad838b84b8007
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:14:57.714Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_authors._id: 65ea7f05e36ad838b84b8008
paper_authors.name: Banghua Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:06:52.744Z
paper_authors._id: 647b8885aba7062fe5c32000
paper_authors.avatarUrl: /avatars/128ceae78490110ae41202851e84d58e.svg
paper_authors.fullname: Banghua Zhu
paper_authors.type: user
paper_authors.user: banghua
paper_authors._id: 65ea7f05e36ad838b84b8009
paper_authors.name: Michael Jordan
paper_authors._id: 65ea7f05e36ad838b84b800a
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:07:21.899Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 65ea7f05e36ad838b84b800b
paper_authors.name: Ion Stoica
paper_title: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
paper_summary: Large Language Models (LLMs) have unlocked new capabilities and applications;
  however, evaluating the alignment with human preferences still poses
  significant challenges. To address this issue, we introduce Chatbot Arena, an
  open platform for evaluating LLMs based on human preferences. Our methodology
  employs a pairwise comparison approach and leverages input from a diverse user
  base through crowdsourcing. The platform has been operational for several
  months, amassing over 240K votes. This paper describes the platform, analyzes
  the data we have collected so far, and explains the tried-and-true statistical
  methods we are using for efficient and accurate evaluation and ranking of
  models. We confirm that the crowdsourced questions are sufficiently diverse and
  discriminating and that the crowdsourced human votes are in good agreement with
  those of expert raters. These analyses collectively establish a robust
  foundation for the credibility of Chatbot Arena. Because of its unique value
  and openness, Chatbot Arena has emerged as one of the most referenced LLM
  leaderboards, widely cited by leading LLM developers and companies. Our demo is
  publicly available at https://chat.lmsys.org.
paper_ai_keywords: Chatbot Arena Large Language Models (LLMs) human preferences pairwise comparison crowdsourcing statistical methods evaluation ranking expert raters