title: Judging the Judges: A Collection of LLM-Generated Relevance Judgements
summary: Using Large Language Models (LLMs) for relevance assessments offers promising
  opportunities to improve Information Retrieval (IR), Natural Language
  Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
  IR experimenters to build evaluation collections with a fraction of the manual
  human labor currently required. This could help with fresh topics on which
  there is still limited knowledge and could mitigate the challenges of
  evaluating ranking systems in low-resource scenarios, where it is challenging
  to find human annotators. Given the fast-paced recent developments in the
  domain, many questions concerning LLMs as assessors are yet to be answered.
  Among the aspects that require further investigation, we can list the impact of
  various components in a relevance judgment generation pipeline, such as the
  prompt used or the LLM chosen.
    This paper benchmarks and reports on the results of a large-scale automatic
  relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
  different relevance assessment approaches were proposed. In detail, we release
  and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
  relevance judgments produced by eight international teams who participated in
  the challenge. Given their diverse nature, these automatically generated
  relevance judgments can help the community not only investigate systematic
  biases caused by LLMs but also explore the effectiveness of ensemble models,
  analyze the trade-offs between different models and human assessors, and
  advance methodologies for improving automated evaluation techniques. The
  released resource is available at the following link:
  https://llm4eval.github.io/LLMJudge-benchmark/
paper_authors._id: 67b75ce1fedef65ff99cf5f8
paper_authors.name: Hossein A. Rahmani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T16:57:36.417Z
paper_authors._id: 64108fc514215c0775e13f5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64108fc514215c0775e13f5e/pHWr8TlBnYrulo2owIrrv.jpeg
paper_authors.fullname: Hossein A. (Saeed) Rahmani
paper_authors.type: user
paper_authors.user: rahmanidashti
paper_authors._id: 67b75ce1fedef65ff99cf5f9
paper_authors.name: Clemencia Siro
paper_authors._id: 67b75ce1fedef65ff99cf5fa
paper_authors.name: Mohammad Aliannejadi
paper_authors._id: 67b75ce1fedef65ff99cf5fb
paper_authors.name: Nick Craswell
paper_authors._id: 67b75ce1fedef65ff99cf5fc
paper_authors.name: Charles L. A. Clarke
paper_authors._id: 67b75ce1fedef65ff99cf5fd
paper_authors.name: Guglielmo Faggioli
paper_authors._id: 67b75ce1fedef65ff99cf5fe
paper_authors.name: Bhaskar Mitra
paper_authors._id: 67b75ce1fedef65ff99cf5ff
paper_authors.name: Paul Thomas
paper_authors._id: 67b75ce1fedef65ff99cf600
paper_authors.name: Emine Yilmaz
paper_title: Judging the Judges: A Collection of LLM-Generated Relevance Judgements
paper_summary: Using Large Language Models (LLMs) for relevance assessments offers promising
  opportunities to improve Information Retrieval (IR), Natural Language
  Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
  IR experimenters to build evaluation collections with a fraction of the manual
  human labor currently required. This could help with fresh topics on which
  there is still limited knowledge and could mitigate the challenges of
  evaluating ranking systems in low-resource scenarios, where it is challenging
  to find human annotators. Given the fast-paced recent developments in the
  domain, many questions concerning LLMs as assessors are yet to be answered.
  Among the aspects that require further investigation, we can list the impact of
  various components in a relevance judgment generation pipeline, such as the
  prompt used or the LLM chosen.
    This paper benchmarks and reports on the results of a large-scale automatic
  relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
  different relevance assessment approaches were proposed. In detail, we release
  and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
  relevance judgments produced by eight international teams who participated in
  the challenge. Given their diverse nature, these automatically generated
  relevance judgments can help the community not only investigate systematic
  biases caused by LLMs but also explore the effectiveness of ensemble models,
  analyze the trade-offs between different models and human assessors, and
  advance methodologies for improving automated evaluation techniques. The
  released resource is available at the following link:
  https://llm4eval.github.io/LLMJudge-benchmark/
paper_ai_keywords: Large Language Models (LLMs) Information Retrieval (IR) Natural Language Processing (NLP) Transformers relevance judgments evaluation collections human labor ranking systems low-resource scenarios annotators scheduling relevance assessment automatic evaluation prompt TREC(track) SIGIR relevance judgments systematic biases ensemble models automated evaluation techniques