title: Collapse of Dense Retrievers: Short, Early, and Literal Biases
    Outranking Factual Evidence
summary: Dense retrieval models are commonly used in Information Retrieval (IR)
  applications, such as Retrieval-Augmented Generation (RAG). Since they often
  serve as the first step in these systems, their robustness is critical to avoid
  failures. In this work, by repurposing a relation extraction dataset (e.g.
  Re-DocRED), we design controlled experiments to quantify the impact of
  heuristic biases, such as favoring shorter documents, in retrievers like
  Dragon+ and Contriever. Our findings reveal significant vulnerabilities:
  retrievers often rely on superficial patterns like over-prioritizing document
  beginnings, shorter documents, repeated entities, and literal matches.
  Additionally, they tend to overlook whether the document contains the query's
  answer, lacking deep semantic understanding. Notably, when multiple biases
  combine, models exhibit catastrophic performance degradation, selecting the
  answer-containing document in less than 3% of cases over a biased document
  without the answer. Furthermore, we show that these biases have direct
  consequences for downstream applications like RAG, where retrieval-preferred
  documents can mislead LLMs, resulting in a 34% performance drop than not
  providing any documents at all.
paper_authors._id: 67d0c4be13d767fbbfaffe94
paper_authors.name: Mohsen Fayyaz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T22:03:29.044Z
paper_authors._id: 60465ecd4f0c980fda768da7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648301912044-60465ecd4f0c980fda768da7.jpeg
paper_authors.fullname: Mohsen Fayyaz
paper_authors.type: user
paper_authors.user: mohsenfayyaz
paper_authors._id: 67d0c4be13d767fbbfaffe95
paper_authors.name: Ali Modarressi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T20:46:08.879Z
paper_authors._id: 60e4738a8c0ddd18fc27ff88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e4738a8c0ddd18fc27ff88/lpLeeIW8r85RTY4fGZTva.jpeg
paper_authors.fullname: Ali Modarressi
paper_authors.type: user
paper_authors.user: amodaresi
paper_authors._id: 67d0c4be13d767fbbfaffe96
paper_authors.name: Hinrich Schuetze
paper_authors._id: 67d0c4be13d767fbbfaffe97
paper_authors.name: Nanyun Peng
paper_title: Collapse of Dense Retrievers: Short, Early, and Literal Biases
    Outranking Factual Evidence
paper_summary: Dense retrieval models are commonly used in Information Retrieval (IR)
  applications, such as Retrieval-Augmented Generation (RAG). Since they often
  serve as the first step in these systems, their robustness is critical to avoid
  failures. In this work, by repurposing a relation extraction dataset (e.g.
  Re-DocRED), we design controlled experiments to quantify the impact of
  heuristic biases, such as favoring shorter documents, in retrievers like
  Dragon+ and Contriever. Our findings reveal significant vulnerabilities:
  retrievers often rely on superficial patterns like over-prioritizing document
  beginnings, shorter documents, repeated entities, and literal matches.
  Additionally, they tend to overlook whether the document contains the query's
  answer, lacking deep semantic understanding. Notably, when multiple biases
  combine, models exhibit catastrophic performance degradation, selecting the
  answer-containing document in less than 3% of cases over a biased document
  without the answer. Furthermore, we show that these biases have direct
  consequences for downstream applications like RAG, where retrieval-preferred
  documents can mislead LLMs, resulting in a 34% performance drop than not
  providing any documents at all.
paper_ai_keywords: Dense retrieval models Information Retrieval (IR) Retrieval-Augmented Generation (RAG) control experiments heuristic biases favoring shorter documents retrievers Dragon+ Contriever superficial patterns over-prioritizing document beginnings repeated entities literal matches deep semantic understanding performance degradation retrieval-preferred documents LLMs performance drop