Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers
<|reference_start|>Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers: The most recent pointwise Large Language Model (LLM) rankers have achieved remarkable ranking results. However, these rankers are hindered by two major drawbacks: (1) they fail to follow a standardized comparison guidance during the ranking process, and (2) they struggle with comprehensive considerations when dealing with complicated passages. To address these shortcomings, we propose to build a ranker that generates ranking scores based on a set of criteria from various perspectives. These criteria are intended to direct each perspective in providing a distinct yet synergistic evaluation. Our research, which examines eight datasets from the BEIR benchmark demonstrates that incorporating this multi-perspective criteria ensemble approach markedly enhanced the performance of pointwise LLM rankers.<|reference_end|>
@article{guo2024generating,
    title={Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers},
    author={Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Qi Zhu, Le Yan,
  Yue Zhang},
    journal={arXiv preprint arXiv:2404.11960},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.11960},
    primaryClass={cs.IR cs.AI}
}