Drowning in Documents: Consequences of Scaling Reranker Inference
Rerankers, typically cross-encoders, are often used to re-score the documents
retrieved by cheaper initial IR systems. This is because, though expensive,
rerankers are assumed to be more effective. We challenge this assumption by
measuring reranker performance for full retrieval, not just re-scoring
first-stage retrieval. Our experiments reveal a surprising trend: the best
existing rerankers provide diminishing returns when scoring progressively more
documents and actually degrade quality beyond a certain limit. In fact, in this
setting, rerankers can frequently assign high scores to documents with no
lexical or semantic overlap with the query. We hope that our findings will spur
future research to improve reranking.
673c0575955070e4cfbc9570
Mathew Jacob
admin_assigned
2024-11-19T15:24:22.335Z
649949e057368427491c3b7d
https://cdn-avatars.huggingface.co/v1/production/uploads/649949e057368427491c3b7d/jxX_8l7XB0rYlkNB_YBGM.jpeg
Jacob
user
MathewJacob
673c0575955070e4cfbc9571
Erik Lindgren
673c0575955070e4cfbc9572
Matei Zaharia
admin_assigned
2024-11-19T15:24:53.605Z
642509c6d476e4ad5568dd31
https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
Matei Zaharia
user
mateiz
673c0575955070e4cfbc9573
Michael Carbin
admin_assigned
2024-11-19T15:24:45.070Z
639b4452c522f98dd1fed3d3
/avatars/f11a8b69b34f8219eb63f914023e8bd1.svg
Michael Carbin
user
mcarbin-mosaicml
673c0575955070e4cfbc9574
Omar Khattab
admin_assigned
2024-11-19T15:25:41.417Z
5f764762f1e7ef6e919a1e0f
/avatars/9445f6efa3f78a870e338988d9ce5573.svg
Omar Khattab
user
okhattab
673c0575955070e4cfbc9575
Andrew Drozdov
extracted_confirmed
2024-11-19T03:58:32.721Z
613fbc22be1df5145babcad0
/avatars/2048e58c43997a60e2dabcfc13112eec.svg
Andrew Drozdov
user
mrdrozdov
Drowning in Documents: Consequences of Scaling Reranker Inference
Rerankers, typically cross-encoders, are often used to re-score the documents
retrieved by cheaper initial IR systems. This is because, though expensive,
rerankers are assumed to be more effective. We challenge this assumption by
measuring reranker performance for full retrieval, not just re-scoring
first-stage retrieval. Our experiments reveal a surprising trend: the best
existing rerankers provide diminishing returns when scoring progressively more
documents and actually degrade quality beyond a certain limit. In fact, in this
setting, rerankers can frequently assign high scores to documents with no
lexical or semantic overlap with the query. We hope that our findings will spur
future research to improve reranking.
Rerankers, typically cross-encoders, offer diminishing returns and degrade quality when scoring a large number of documents, challenging the assumption that they are consistently more effective.
rerankers cross-encoders initial IR systems full retrieval re-scoring semantic overlap