title: Drowning in Documents: Consequences of Scaling Reranker Inference
summary: Rerankers, typically cross-encoders, are often used to re-score the documents
  retrieved by cheaper initial IR systems. This is because, though expensive,
  rerankers are assumed to be more effective. We challenge this assumption by
  measuring reranker performance for full retrieval, not just re-scoring
  first-stage retrieval. Our experiments reveal a surprising trend: the best
  existing rerankers provide diminishing returns when scoring progressively more
  documents and actually degrade quality beyond a certain limit. In fact, in this
  setting, rerankers can frequently assign high scores to documents with no
  lexical or semantic overlap with the query. We hope that our findings will spur
  future research to improve reranking.
paper_authors._id: 673c0575955070e4cfbc9570
paper_authors.name: Mathew Jacob
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:24:22.335Z
paper_authors._id: 649949e057368427491c3b7d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649949e057368427491c3b7d/jxX_8l7XB0rYlkNB_YBGM.jpeg
paper_authors.fullname: Jacob
paper_authors.type: user
paper_authors.user: MathewJacob
paper_authors._id: 673c0575955070e4cfbc9571
paper_authors.name: Erik Lindgren
paper_authors._id: 673c0575955070e4cfbc9572
paper_authors.name: Matei Zaharia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:24:53.605Z
paper_authors._id: 642509c6d476e4ad5568dd31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
paper_authors.fullname: Matei Zaharia
paper_authors.type: user
paper_authors.user: mateiz
paper_authors._id: 673c0575955070e4cfbc9573
paper_authors.name: Michael Carbin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:24:45.070Z
paper_authors._id: 639b4452c522f98dd1fed3d3
paper_authors.avatarUrl: /avatars/f11a8b69b34f8219eb63f914023e8bd1.svg
paper_authors.fullname: Michael Carbin
paper_authors.type: user
paper_authors.user: mcarbin-mosaicml
paper_authors._id: 673c0575955070e4cfbc9574
paper_authors.name: Omar Khattab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:25:41.417Z
paper_authors._id: 5f764762f1e7ef6e919a1e0f
paper_authors.avatarUrl: /avatars/9445f6efa3f78a870e338988d9ce5573.svg
paper_authors.fullname: Omar Khattab
paper_authors.type: user
paper_authors.user: okhattab
paper_authors._id: 673c0575955070e4cfbc9575
paper_authors.name: Andrew Drozdov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-19T03:58:32.721Z
paper_authors._id: 613fbc22be1df5145babcad0
paper_authors.avatarUrl: /avatars/2048e58c43997a60e2dabcfc13112eec.svg
paper_authors.fullname: Andrew Drozdov
paper_authors.type: user
paper_authors.user: mrdrozdov
paper_title: Drowning in Documents: Consequences of Scaling Reranker Inference
paper_summary: Rerankers, typically cross-encoders, are often used to re-score the documents
  retrieved by cheaper initial IR systems. This is because, though expensive,
  rerankers are assumed to be more effective. We challenge this assumption by
  measuring reranker performance for full retrieval, not just re-scoring
  first-stage retrieval. Our experiments reveal a surprising trend: the best
  existing rerankers provide diminishing returns when scoring progressively more
  documents and actually degrade quality beyond a certain limit. In fact, in this
  setting, rerankers can frequently assign high scores to documents with no
  lexical or semantic overlap with the query. We hope that our findings will spur
  future research to improve reranking.
paper_ai_keywords: cross-encoders rerankers initial IR systems re-scoring first-stage retrieval diminishing returns lexical semantic overlap query