title: Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
summary: Evaluating large language model (LLM) based chat assistants is challenging
  due to their broad capabilities and the inadequacy of existing benchmarks in
  measuring human preferences. To address this, we explore using strong LLMs as
  judges to evaluate these models on more open-ended questions. We examine the
  usage and limitations of LLM-as-a-judge, such as position and verbosity biases
  and limited reasoning ability, and propose solutions to migrate some of them.
  We then verify the agreement between LLM judges and human preferences by
  introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot
  Arena, a crowdsourced battle platform. Our results reveal that strong LLM
  judges like GPT-4 can match both controlled and crowdsourced human preferences
  well, achieving over 80\% agreement, the same level of agreement between
  humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate
  human preferences, which are otherwise very expensive to obtain. Additionally,
  we show our benchmark and traditional benchmarks complement each other by
  evaluating several variants of LLaMA/Vicuna. We will publicly release 80
  MT-bench questions, 3K expert votes, and 30K conversations with human
  preferences from Chatbot Arena.
paper_authors._id: 64866c1360b30c3615c7d12a
paper_authors.name: Lianmin Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T18:59:04.900Z
paper_authors._id: 62d35f3ceaf3858ce253ab7a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d35f3ceaf3858ce253ab7a/3yiiMriltXJBi242FZY-W.jpeg
paper_authors.fullname: Lianmin
paper_authors.type: user
paper_authors.user: lmzheng
paper_authors._id: 64866c1360b30c3615c7d12b
paper_authors.name: Wei-Lin Chiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T19:24:36.433Z
paper_authors._id: 63071772fd79b417f1b92291
paper_authors.avatarUrl: /avatars/ec68d20dcedf4553bb31d9f6e0ded813.svg
paper_authors.fullname: Wei-Lin Chiang
paper_authors.type: user
paper_authors.user: weichiang
paper_authors._id: 64866c1360b30c3615c7d12c
paper_authors.name: Ying Sheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:46:25.519Z
paper_authors._id: 6455b969654d8bccae50736c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
paper_authors.fullname: Ying Sheng
paper_authors.type: user
paper_authors.user: ying1123
paper_authors._id: 64866c1360b30c3615c7d12d
paper_authors.name: Siyuan Zhuang
paper_authors._id: 64866c1360b30c3615c7d12e
paper_authors.name: Zhanghao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:46:01.420Z
paper_authors._id: 6429b84e8852afdf89beeae5
paper_authors.avatarUrl: /avatars/7cfc1a484077a8d511c46392de8dee07.svg
paper_authors.fullname: Zhanghao Wu
paper_authors.type: user
paper_authors.user: michaelvll
paper_authors._id: 64866c1360b30c3615c7d12f
paper_authors.name: Yonghao Zhuang
paper_authors._id: 64866c1360b30c3615c7d130
paper_authors.name: Zi Lin
paper_authors._id: 64866c1360b30c3615c7d131
paper_authors.name: Zhuohan Li
paper_authors._id: 64866c1360b30c3615c7d132
paper_authors.name: Dacheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:47:12.747Z
paper_authors._id: 63715b25ffc0489ed7d1f415
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
paper_authors.fullname: Dacheng Li
paper_authors.type: user
paper_authors.user: DachengLi
paper_authors._id: 64866c1360b30c3615c7d133
paper_authors.name: Eric. P Xing
paper_authors._id: 64866c1360b30c3615c7d134
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:15:17.449Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_authors._id: 64866c1360b30c3615c7d135
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 64866c1360b30c3615c7d136
paper_authors.name: Ion Stoica
paper_title: Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
paper_summary: Evaluating large language model (LLM) based chat assistants is challenging
  due to their broad capabilities and the inadequacy of existing benchmarks in
  measuring human preferences. To address this, we explore using strong LLMs as
  judges to evaluate these models on more open-ended questions. We examine the
  usage and limitations of LLM-as-a-judge, such as position and verbosity biases
  and limited reasoning ability, and propose solutions to migrate some of them.
  We then verify the agreement between LLM judges and human preferences by
  introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot
  Arena, a crowdsourced battle platform. Our results reveal that strong LLM
  judges like GPT-4 can match both controlled and crowdsourced human preferences
  well, achieving over 80\% agreement, the same level of agreement between
  humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate
  human preferences, which are otherwise very expensive to obtain. Additionally,
  we show our benchmark and traditional benchmarks complement each other by
  evaluating several variants of LLaMA/Vicuna. We will publicly release 80
  MT-bench questions, 3K expert votes, and 30K conversations with human
  preferences from Chatbot Arena.
paper_ai_keywords: LLM large language model chat assistants benchmarks human preferences LLM-as-a-judge position and verbosity biases limited reasoning ability MT-bench multi-turn question set Chatbot Arena crowdsourced battle platform GPT-4 downstream tasks LLaMA Vicuna expert votes conversations with human preferences