title: Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard
    Negatives for Robust Information Retrieval
summary: Training robust retrieval and reranker models typically relies on large-scale
  retrieval datasets; for example, the BGE collection contains 1.6 million
  query-passage pairs sourced from various data sources. However, we find that
  certain datasets can negatively impact model effectiveness -- pruning 8 out of
  15 datasets from the BGE collection reduces the training set size by
  2.35times and increases nDCG@10 on BEIR by 1.0 point. This motivates a
  deeper examination of training data quality, with a particular focus on "false
  negatives", where relevant passages are incorrectly labeled as irrelevant. We
  propose a simple, cost-effective approach using cascading LLM prompts to
  identify and relabel hard negatives. Experimental results show that relabeling
  false negatives with true positives improves both E5 (base) and Qwen2.5-7B
  retrieval models by 0.7-1.4 nDCG@10 on BEIR and by 1.7-1.8 nDCG@10 on zero-shot
  AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on
  the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of the
  cascading design is further supported by human annotation results, where we
  find judgment by GPT-4o shows much higher agreement with humans than
  GPT-4o-mini.
paper_authors._id: 683049bd842b92383b946392
paper_authors.name: Nandan Thakur
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:14:21.488Z
paper_authors._id: 60196690dd31fde3c1062960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
paper_authors.fullname: Nandan Thakur
paper_authors.type: user
paper_authors.user: nthakur
paper_authors._id: 683049bd842b92383b946393
paper_authors.name: Crystina Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:14:18.843Z
paper_authors._id: 611fb1cbfa8355ed0309de81
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665676377085-611fb1cbfa8355ed0309de81.jpeg
paper_authors.fullname: Xinyu ZHANG
paper_authors.type: user
paper_authors.user: crystina-z
paper_authors._id: 683049bd842b92383b946394
paper_authors.name: Xueguang Ma
paper_authors._id: 683049bd842b92383b946395
paper_authors.name: Jimmy Lin
paper_title: Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard
    Negatives for Robust Information Retrieval
paper_summary: Training robust retrieval and reranker models typically relies on large-scale
  retrieval datasets; for example, the BGE collection contains 1.6 million
  query-passage pairs sourced from various data sources. However, we find that
  certain datasets can negatively impact model effectiveness -- pruning 8 out of
  15 datasets from the BGE collection reduces the training set size by
  2.35times and increases nDCG@10 on BEIR by 1.0 point. This motivates a
  deeper examination of training data quality, with a particular focus on "false
  negatives", where relevant passages are incorrectly labeled as irrelevant. We
  propose a simple, cost-effective approach using cascading LLM prompts to
  identify and relabel hard negatives. Experimental results show that relabeling
  false negatives with true positives improves both E5 (base) and Qwen2.5-7B
  retrieval models by 0.7-1.4 nDCG@10 on BEIR and by 1.7-1.8 nDCG@10 on zero-shot
  AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on
  the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of the
  cascading design is further supported by human annotation results, where we
  find judgment by GPT-4o shows much higher agreement with humans than
  GPT-4o-mini.
paper_ai_summary: Using cascading LLM prompts to identify and relabel false negatives in datasets improves retrieval and reranking models' performance.
paper_ai_keywords: retrieval models reranker models BGE collection nDCG@10 BEIR false negatives E5 (base) Qwen2.5-7B zero-shot AIR-Bench GPT-4o GPT-4o-mini