title: Are LLMs Better than Reported? Detecting Label Errors and Mitigating
    Their Effect on Model Performance
summary: NLP benchmarks rely on standardized datasets for training and evaluating
  models and are crucial for advancing the field. Traditionally, expert
  annotations ensure high-quality labels; however, the cost of expert annotation
  does not scale well with the growing demand for larger datasets required by
  modern models. While crowd-sourcing provides a more scalable solution, it often
  comes at the expense of annotation precision and consistency. Recent
  advancements in large language models (LLMs) offer new opportunities to enhance
  the annotation process, particularly for detecting label errors in existing
  datasets. In this work, we consider the recent approach of LLM-as-a-judge,
  leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through
  a case study of four datasets from the TRUE benchmark, covering different tasks
  and domains, we empirically analyze the labeling quality of existing datasets,
  and compare expert, crowd-sourced, and our LLM-based annotations in terms of
  agreement, label quality, and efficiency, demonstrating the strengths and
  limitations of each annotation method. Our findings reveal a substantial number
  of label errors, which, when corrected, induce a significant upward shift in
  reported model performance. This suggests that many of the LLMs so-called
  mistakes are due to label errors rather than genuine model failures.
  Additionally, we discuss the implications of mislabeled data and propose
  methods to mitigate them in training to improve model performance.
paper_authors._id: 671f8971915aa3d2d1691b9b
paper_authors.name: Omer Nahum
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:37:31.730Z
paper_authors._id: 663511a0efbee9a1fde43940
paper_authors.avatarUrl: /avatars/ad124ac100829359f67052c730e0a0d4.svg
paper_authors.fullname: Omer Nahum
paper_authors.type: user
paper_authors.user: omer6nahum
paper_authors._id: 671f8971915aa3d2d1691b9c
paper_authors.name: Nitay Calderon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:37:37.337Z
paper_authors._id: 62d6a0c18faee0ac953c51fa
paper_authors.avatarUrl: /avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
paper_authors.fullname: Nitay Calderon
paper_authors.type: user
paper_authors.user: nitay
paper_authors._id: 671f8971915aa3d2d1691b9d
paper_authors.name: Orgad Keller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:37:42.433Z
paper_authors._id: 66f00fe216bffdccd51832e2
paper_authors.avatarUrl: /avatars/6d2330d2353ed40cbff6bd4a71fb7865.svg
paper_authors.fullname: Orgad Keller
paper_authors.type: user
paper_authors.user: orgad
paper_authors._id: 671f8971915aa3d2d1691b9e
paper_authors.name: Idan Szpektor
paper_authors._id: 671f8971915aa3d2d1691b9f
paper_authors.name: Roi Reichart
paper_title: Are LLMs Better than Reported? Detecting Label Errors and Mitigating
    Their Effect on Model Performance
paper_summary: NLP benchmarks rely on standardized datasets for training and evaluating
  models and are crucial for advancing the field. Traditionally, expert
  annotations ensure high-quality labels; however, the cost of expert annotation
  does not scale well with the growing demand for larger datasets required by
  modern models. While crowd-sourcing provides a more scalable solution, it often
  comes at the expense of annotation precision and consistency. Recent
  advancements in large language models (LLMs) offer new opportunities to enhance
  the annotation process, particularly for detecting label errors in existing
  datasets. In this work, we consider the recent approach of LLM-as-a-judge,
  leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through
  a case study of four datasets from the TRUE benchmark, covering different tasks
  and domains, we empirically analyze the labeling quality of existing datasets,
  and compare expert, crowd-sourced, and our LLM-based annotations in terms of
  agreement, label quality, and efficiency, demonstrating the strengths and
  limitations of each annotation method. Our findings reveal a substantial number
  of label errors, which, when corrected, induce a significant upward shift in
  reported model performance. This suggests that many of the LLMs so-called
  mistakes are due to label errors rather than genuine model failures.
  Additionally, we discuss the implications of mislabeled data and propose
  methods to mitigate them in training to improve model performance.
paper_ai_summary: LLMs used as judges can identify label errors in datasets, improving annotation quality and model performance across various tasks and domains.
paper_ai_keywords: large language models LLM-as-a-judge ensemble models TRUE benchmark labeling quality annotation efficiency