A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE
<|reference_start|>A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE: We present a comparative study between cross-encoder and LLMs rerankers in the context of re-ranking effective SPLADE retrievers. We conduct a large evaluation on TREC Deep Learning datasets and out-of-domain datasets such as BEIR and LoTTE. In the first set of experiments, we show how cross-encoder rerankers are hard to distinguish when it comes to re-rerank SPLADE on MS MARCO. Observations shift in the out-of-domain scenario, where both the type of model and the number of documents to re-rank have an impact on effectiveness. Then, we focus on listwise rerankers based on Large Language Models -- especially GPT-4. While GPT-4 demonstrates impressive (zero-shot) performance, we show that traditional cross-encoders remain very competitive. Overall, our findings aim to to provide a more nuanced perspective on the recent excitement surrounding LLM-based re-rankers -- by positioning them as another factor to consider in balancing effectiveness and efficiency in search systems.<|reference_end|>
@article{déjean2024a,
    title={A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE},
    author={Herv'e D'ejean, St'ephane Clinchant, Thibault Formal},
    journal={arXiv preprint arXiv:2403.10407},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10407},
    primaryClass={cs.IR}
}