title: JuStRank: Benchmarking LLM Judges for System Ranking
summary: Given the rapid progress of generative AI, there is a pressing need to
  systematically compare and choose between the numerous models and
  configurations available. The scale and versatility of such evaluations make
  the use of LLM-based judges a compelling solution for this challenge.
  Crucially, this approach requires first to validate the quality of the LLM
  judge itself. Previous work has focused on instance-based assessment of LLM
  judges, where a judge is evaluated over a set of responses, or response pairs,
  while being agnostic to their source systems. We argue that this setting
  overlooks critical factors affecting system-level ranking, such as a judge's
  positive or negative bias towards certain systems. To address this gap, we
  conduct the first large-scale study of LLM judges as system rankers. System
  scores are generated by aggregating judgment scores over multiple system
  outputs, and the judge's quality is assessed by comparing the resulting system
  ranking to a human-based ranking. Beyond overall judge assessment, our analysis
  provides a fine-grained characterization of judge behavior, including their
  decisiveness and bias.
paper_authors._id: 675bdf1ba301c0203a58d39d
paper_authors.name: Ariel Gera
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:15:38.275Z
paper_authors._id: 6415ebd2107962562e9a0712
paper_authors.avatarUrl: /avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
paper_authors.fullname: Ariel Gera
paper_authors.type: user
paper_authors.user: arielgera
paper_authors._id: 675bdf1ba301c0203a58d39e
paper_authors.name: Odellia Boni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:21:02.044Z
paper_authors._id: 6742ef68f6646c6a4e0488e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/06egZhjicM7TJ4gXvW8yH.png
paper_authors.fullname: odellia boni
paper_authors.type: user
paper_authors.user: odelliab
paper_authors._id: 675bdf1ba301c0203a58d39f
paper_authors.name: Yotam Perlitz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:53:33.385Z
paper_authors._id: 61e01a92a24b774dac1f32d7
paper_authors.avatarUrl: /avatars/307aac0c6cfe1879f6a5e5e1d6c10413.svg
paper_authors.fullname: Yotam Perlitz
paper_authors.type: user
paper_authors.user: per
paper_authors._id: 675bdf1ba301c0203a58d3a0
paper_authors.name: Roy Bar-Haim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:52.572Z
paper_authors._id: 666d8753e70e5838d9716446
paper_authors.avatarUrl: /avatars/6390f9069372a72e79cd331bf043908b.svg
paper_authors.fullname: Roy Bar Haim
paper_authors.type: user
paper_authors.user: roybarhaim
paper_authors._id: 675bdf1ba301c0203a58d3a1
paper_authors.name: Lilach Eden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:47.110Z
paper_authors._id: 6694bdae1a0138cdbfc2799b
paper_authors.avatarUrl: /avatars/8c42646280c327253493edc149d16a56.svg
paper_authors.fullname: Lilach Eden
paper_authors.type: user
paper_authors.user: LilachE
paper_authors._id: 675bdf1ba301c0203a58d3a2
paper_authors.name: Asaf Yehudai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:06.387Z
paper_authors._id: 638324f862badff43269e588
paper_authors.avatarUrl: /avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
paper_authors.fullname: Asaf Yehudai
paper_authors.type: user
paper_authors.user: Asaf-Yehudai
paper_title: JuStRank: Benchmarking LLM Judges for System Ranking
paper_summary: Given the rapid progress of generative AI, there is a pressing need to
  systematically compare and choose between the numerous models and
  configurations available. The scale and versatility of such evaluations make
  the use of LLM-based judges a compelling solution for this challenge.
  Crucially, this approach requires first to validate the quality of the LLM
  judge itself. Previous work has focused on instance-based assessment of LLM
  judges, where a judge is evaluated over a set of responses, or response pairs,
  while being agnostic to their source systems. We argue that this setting
  overlooks critical factors affecting system-level ranking, such as a judge's
  positive or negative bias towards certain systems. To address this gap, we
  conduct the first large-scale study of LLM judges as system rankers. System
  scores are generated by aggregating judgment scores over multiple system
  outputs, and the judge's quality is assessed by comparing the resulting system
  ranking to a human-based ranking. Beyond overall judge assessment, our analysis
  provides a fine-grained characterization of judge behavior, including their
  decisiveness and bias.
paper_ai_summary: A study evaluates LLM judges for system-level ranking in generative AI, assessing their quality, decisiveness, and bias through comparison with human-based rankings.
paper_ai_keywords: LLM-based judges system-level ranking instance-based assessment judge quality human-based ranking decisiveness bias