title: WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in
    the Wild
summary: We introduce WildBench, an automated evaluation framework designed to
  benchmark large language models (LLMs) using challenging, real-world user
  queries. WildBench consists of 1,024 tasks carefully selected from over one
  million human-chatbot conversation logs. For automated evaluation with
  WildBench, we have developed two metrics, WB-Reward and WB-Score, which are
  computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses
  task-specific checklists to evaluate model outputs systematically and provides
  structured explanations that justify the scores and comparisons, resulting in
  more reliable and interpretable automatic judgments. WB-Reward employs
  fine-grained pairwise comparisons between model responses, generating five
  potential outcomes: much better, slightly better, slightly worse, much worse,
  or a tie. Unlike previous evaluations that employed a single baseline model, we
  selected three baseline models at varying performance levels to ensure a
  comprehensive pairwise evaluation. Additionally, we propose a simple method to
  mitigate length bias, by converting outcomes of ``slightly better/worse'' to
  ``tie'' if the winner response exceeds the loser one by more than K
  characters. WB-Score evaluates the quality of model outputs individually,
  making it a fast and cost-efficient evaluation metric. WildBench results
  demonstrate a strong correlation with the human-voted Elo ratings from Chatbot
  Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of
  0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing
  both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates,
  as well as the 0.87 for regular win rates.
paper_authors._id: 66665de49f5a0b322905de64
paper_authors.name: Bill Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:19:45.095Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 66665de49f5a0b322905de65
paper_authors.name: Yuntian Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:19:52.064Z
paper_authors._id: 63081e15a670ed10f9d44229
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
paper_authors.fullname: Yuntian Deng
paper_authors.type: user
paper_authors.user: yuntian-deng
paper_authors._id: 66665de49f5a0b322905de66
paper_authors.name: Khyathi Chandu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:20:22.751Z
paper_authors._id: 62ebe6472f8999993a847bb6
paper_authors.avatarUrl: /avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
paper_authors.fullname: Khyathi Raghavi Chandu
paper_authors.type: user
paper_authors.user: khyathi
paper_authors._id: 66665de49f5a0b322905de67
paper_authors.name: Faeze Brahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:20:29.009Z
paper_authors._id: 65282b8d578679aac7888aec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
paper_authors.fullname: Faeze Brahman
paper_authors.type: user
paper_authors.user: faezeb
paper_authors._id: 66665de49f5a0b322905de68
paper_authors.name: Abhilasha Ravichander
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:20:49.502Z
paper_authors._id: 6349886c429608888c42319a
paper_authors.avatarUrl: /avatars/f84b5fe8b76172878274754e3399d6ec.svg
paper_authors.fullname: Abhilasha Ravichander
paper_authors.type: user
paper_authors.user: lasha-nlp
paper_authors._id: 66665de49f5a0b322905de69
paper_authors.name: Valentina Pyatkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:20:56.401Z
paper_authors._id: 6556cff80e7a7067a934445f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
paper_authors.fullname: Valentina Pyatkin
paper_authors.type: user
paper_authors.user: valpy
paper_authors._id: 66665de49f5a0b322905de6a
paper_authors.name: Nouha Dziri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:21:19.482Z
paper_authors._id: 625888c2c964680d85d70bff
paper_authors.avatarUrl: /avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhadziri
paper_authors._id: 66665de49f5a0b322905de6b
paper_authors.name: Ronan Le Bras
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:21:26.252Z
paper_authors._id: 635049104e753c9940fefd71
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
paper_authors.fullname: Ronan Le Bras
paper_authors.type: user
paper_authors.user: ronanlb
paper_authors._id: 66665de49f5a0b322905de6c
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-10T10:21:33.126Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in
    the Wild
paper_summary: We introduce WildBench, an automated evaluation framework designed to
  benchmark large language models (LLMs) using challenging, real-world user
  queries. WildBench consists of 1,024 tasks carefully selected from over one
  million human-chatbot conversation logs. For automated evaluation with
  WildBench, we have developed two metrics, WB-Reward and WB-Score, which are
  computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses
  task-specific checklists to evaluate model outputs systematically and provides
  structured explanations that justify the scores and comparisons, resulting in
  more reliable and interpretable automatic judgments. WB-Reward employs
  fine-grained pairwise comparisons between model responses, generating five
  potential outcomes: much better, slightly better, slightly worse, much worse,
  or a tie. Unlike previous evaluations that employed a single baseline model, we
  selected three baseline models at varying performance levels to ensure a
  comprehensive pairwise evaluation. Additionally, we propose a simple method to
  mitigate length bias, by converting outcomes of ``slightly better/worse'' to
  ``tie'' if the winner response exceeds the loser one by more than K
  characters. WB-Score evaluates the quality of model outputs individually,
  making it a fast and cost-efficient evaluation metric. WildBench results
  demonstrate a strong correlation with the human-voted Elo ratings from Chatbot
  Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of
  0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing
  both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates,
  as well as the 0.87 for regular win rates.
paper_ai_keywords: LLMs (Large Language Models) WildBench GPT-4-turbo WB-Reward WB-Score fine-grained pairwise comparisons length bias WB-Reward WB-Score human-voted Elo ratings Chatbot Arena Pearson correlation ArenaHard AlpacaEval2.0