title: Benchmark Agreement Testing Done Right: A Guide for LLM Benchmark
    Evaluation
summary: Recent advancements in Language Models (LMs) have catalyzed the creation of
  multiple benchmarks, designed to assess these models' general capabilities. A
  crucial task, however, is assessing the validity of the benchmarks themselves.
  This is most commonly done via Benchmark Agreement Testing (BAT), where new
  benchmarks are validated against established ones using some agreement metric
  (e.g., rank correlation). Despite the crucial role of BAT for benchmark
  builders and consumers, there are no standardized procedures for such agreement
  testing. This deficiency can lead to invalid conclusions, fostering mistrust in
  benchmarks and upending the ability to properly choose the appropriate
  benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how
  some overlooked methodological choices can significantly influence BAT results,
  potentially undermining the validity of conclusions. To address these
  inconsistencies, we propose a set of best practices for BAT and demonstrate how
  utilizing these methodologies greatly improves BAT robustness and validity. To
  foster adoption and facilitate future research,, we introduce BenchBench, a
  python package for BAT, and release the BenchBench-leaderboard, a
  meta-benchmark designed to evaluate benchmarks using their peers. Our findings
  underscore the necessity for standardized BAT, ensuring the robustness and
  validity of benchmark evaluations in the evolving landscape of language model
  research.
    BenchBench Package: https://github.com/IBM/BenchBench
    Leaderboard: https://huggingface.co/spaces/per/BenchBench
paper_authors._id: 6699ed99004e418e714b8fbf
paper_authors.name: Yotam Perlitz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:48:20.236Z
paper_authors._id: 61e01a92a24b774dac1f32d7
paper_authors.avatarUrl: /avatars/307aac0c6cfe1879f6a5e5e1d6c10413.svg
paper_authors.fullname: Yotam Perlitz
paper_authors.type: user
paper_authors.user: per
paper_authors._id: 6699ed99004e418e714b8fc0
paper_authors.name: Ariel Gera
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T13:59:22.592Z
paper_authors._id: 6415ebd2107962562e9a0712
paper_authors.avatarUrl: /avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
paper_authors.fullname: Ariel Gera
paper_authors.type: user
paper_authors.user: arielgera
paper_authors._id: 6699ed99004e418e714b8fc1
paper_authors.name: Ofir Arviv
paper_authors._id: 6699ed99004e418e714b8fc2
paper_authors.name: Asaf Yehudai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:43:18.742Z
paper_authors._id: 638324f862badff43269e588
paper_authors.avatarUrl: /avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
paper_authors.fullname: Asaf Yehudai
paper_authors.type: user
paper_authors.user: Asaf-Yehudai
paper_authors._id: 6699ed99004e418e714b8fc3
paper_authors.name: Elron Bandel
paper_authors._id: 6699ed99004e418e714b8fc4
paper_authors.name: Eyal Shnarch
paper_authors._id: 6699ed99004e418e714b8fc5
paper_authors.name: Michal Shmueli-Scheuer
paper_authors._id: 6699ed99004e418e714b8fc6
paper_authors.name: Leshem Choshen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:58:07.984Z
paper_authors._id: 61bf40824b4300d0fb0acf59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
paper_authors.fullname: Leshem Choshen
paper_authors.type: user
paper_authors.user: borgr
paper_title: Benchmark Agreement Testing Done Right: A Guide for LLM Benchmark
    Evaluation
paper_summary: Recent advancements in Language Models (LMs) have catalyzed the creation of
  multiple benchmarks, designed to assess these models' general capabilities. A
  crucial task, however, is assessing the validity of the benchmarks themselves.
  This is most commonly done via Benchmark Agreement Testing (BAT), where new
  benchmarks are validated against established ones using some agreement metric
  (e.g., rank correlation). Despite the crucial role of BAT for benchmark
  builders and consumers, there are no standardized procedures for such agreement
  testing. This deficiency can lead to invalid conclusions, fostering mistrust in
  benchmarks and upending the ability to properly choose the appropriate
  benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how
  some overlooked methodological choices can significantly influence BAT results,
  potentially undermining the validity of conclusions. To address these
  inconsistencies, we propose a set of best practices for BAT and demonstrate how
  utilizing these methodologies greatly improves BAT robustness and validity. To
  foster adoption and facilitate future research,, we introduce BenchBench, a
  python package for BAT, and release the BenchBench-leaderboard, a
  meta-benchmark designed to evaluate benchmarks using their peers. Our findings
  underscore the necessity for standardized BAT, ensuring the robustness and
  validity of benchmark evaluations in the evolving landscape of language model
  research.
    BenchBench Package: https://github.com/IBM/BenchBench
    Leaderboard: https://huggingface.co/spaces/per/BenchBench