A Systematic Investigation of Distilling Large Language Models into Cross-Encoders for Passage Re-ranking
<|reference_start|>A Systematic Investigation of Distilling Large Language Models into Cross-Encoders for Passage Re-ranking: Cross-encoders distilled from large language models (LLMs) are often more effective re-rankers than cross-encoders fine-tuned on manually labeled data. However, the distilled models usually do not reach their teacher LLM's effectiveness. To investigate whether best practices for fine-tuning cross-encoders on manually labeled data (e.g., hard-negative sampling, deep sampling, and listwise loss functions) can help to improve LLM ranker distillation, we construct and release a new distillation dataset: Rank-DistiLLM. In our experiments, cross-encoders trained on Rank-DistiLLM reach the effectiveness of LLMs while being orders of magnitude more efficient. Our code and data is available at https://github.com/webis-de/msmarco-llm-distillation.<|reference_end|>
@article{schlatt2024a,
    title={A Systematic Investigation of Distilling Large Language Models into
  Cross-Encoders for Passage Re-ranking},
    author={Ferdinand Schlatt, Maik Fr"obe, Harrisen Scells, Shengyao Zhuang,
  Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, Matthias Hagen},
    journal={arXiv preprint arXiv:2405.07920},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07920},
    primaryClass={cs.IR}
}