title: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
    Diverse Models
summary: As Large Language Models (LLMs) have become more advanced, they have outpaced
  our abilities to accurately evaluate their quality. Not only is finding data to
  adequately probe particular model properties difficult, but evaluating the
  correctness of a model's freeform generation alone is a challenge. To address
  this, many evaluations now rely on using LLMs themselves as judges to score the
  quality of outputs from other LLMs. Evaluations most commonly use a single
  large model like GPT4. While this method has grown in popularity, it is costly,
  has been shown to introduce intramodel bias, and in this work, we find that
  very large models are often unnecessary. We propose instead to evaluate models
  using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
  and spanning six different datasets, we find that using a PoLL composed of a
  larger number of smaller models outperforms a single large judge, exhibits less
  intra-model bias due to its composition of disjoint model families, and does so
  while being over seven times less expensive.
paper_authors._id: 66305fa1cee11621a131e9a0
paper_authors.name: Pat Verga
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:30:04.908Z
paper_authors._id: 6610180904bc9d80c02be453
paper_authors.avatarUrl: /avatars/9b8bd16a862e5004e907e2af5c18e1db.svg
paper_authors.fullname: Pat Verga
paper_authors.type: user
paper_authors.user: pat-cohere
paper_authors._id: 66305fa1cee11621a131e9a1
paper_authors.name: Sebastian Hofstatter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:30:16.294Z
paper_authors._id: 60506d8a5f211e59bdaff26b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616085391299-60506d8a5f211e59bdaff26b.jpeg
paper_authors.fullname: Sebastian Hofstätter
paper_authors.type: user
paper_authors.user: sebastian-hofstaetter
paper_authors._id: 66305fa1cee11621a131e9a2
paper_authors.name: Sophia Althammer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:30:23.276Z
paper_authors._id: 642c574c53e76b4c22842843
paper_authors.avatarUrl: /avatars/09095a843c5f0dc8af998c893e41cd64.svg
paper_authors.fullname: Sophia Althammer
paper_authors.type: user
paper_authors.user: salthammer
paper_authors._id: 66305fa1cee11621a131e9a3
paper_authors.name: Yixuan Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:34:32.933Z
paper_authors._id: 6144c40da8f4f7baca305243
paper_authors.avatarUrl: /avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
paper_authors.fullname: Yixuan Su
paper_authors.type: user
paper_authors.user: pangpang666
paper_authors._id: 66305fa1cee11621a131e9a4
paper_authors.name: Aleksandra Piktus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:34:40.337Z
paper_authors._id: 5f5f4f093c67af20d9945a6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1600081648206-noauth.jpeg
paper_authors.fullname: Aleksandra Piktus
paper_authors.type: user
paper_authors.user: ola13
paper_authors._id: 66305fa1cee11621a131e9a5
paper_authors.name: Arkady Arkhangorodsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:35:03.090Z
paper_authors._id: 65327ba4c7530aa27f690d32
paper_authors.avatarUrl: /avatars/3baa31fbc9d1b1b2b7ead56ea42eea0b.svg
paper_authors.fullname: Arkady Arkhangorodsky
paper_authors.type: user
paper_authors.user: arkadyark-cohere
paper_authors._id: 66305fa1cee11621a131e9a6
paper_authors.name: Minjie Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:35:12.387Z
paper_authors._id: 63a49568658851481f770bb6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671730818150-63a49568658851481f770bb6.jpeg
paper_authors.fullname: Minjie Xu
paper_authors.type: user
paper_authors.user: chokkyvista
paper_authors._id: 66305fa1cee11621a131e9a7
paper_authors.name: Naomi White
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:35:18.649Z
paper_authors._id: 660e71ef076a625565bca191
paper_authors.avatarUrl: /avatars/0c8df0ea42aad295fb77c2abeba852cc.svg
paper_authors.fullname: Naomi White
paper_authors.type: user
paper_authors.user: naomiwhite
paper_authors._id: 66305fa1cee11621a131e9a8
paper_authors.name: Patrick Lewis
paper_title: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
    Diverse Models
paper_summary: As Large Language Models (LLMs) have become more advanced, they have outpaced
  our abilities to accurately evaluate their quality. Not only is finding data to
  adequately probe particular model properties difficult, but evaluating the
  correctness of a model's freeform generation alone is a challenge. To address
  this, many evaluations now rely on using LLMs themselves as judges to score the
  quality of outputs from other LLMs. Evaluations most commonly use a single
  large model like GPT4. While this method has grown in popularity, it is costly,
  has been shown to introduce intramodel bias, and in this work, we find that
  very large models are often unnecessary. We propose instead to evaluate models
  using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
  and spanning six different datasets, we find that using a PoLL composed of a
  larger number of smaller models outperforms a single large judge, exhibits less
  intra-model bias due to its composition of disjoint model families, and does so
  while being over seven times less expensive.
paper_ai_summary: Evaluating Large Language Models using a Panel of smaller evaluators outperforms single large evaluators with reduced bias and cost.
paper_ai_keywords: Large Language Models LLMs evaluators PoLL model properties freeform generation GPT4 intramodel bias disjoint model families