Large Language Models are Effective Text Rankers with Pairwise Ranking
  Prompting
Ranking documents using Large Language Models (LLMs) by directly feeding the
query and candidate documents into the prompt is an interesting and practical
problem. However, there has been limited success so far, as researchers have
found it difficult to outperform fine-tuned baseline rankers on benchmark
datasets. We analyze pointwise and listwise ranking prompts used by existing
methods and argue that off-the-shelf LLMs do not fully understand these ranking
formulations, possibly due to the nature of how LLMs are trained. In this
paper, we propose to significantly reduce the burden on LLMs by using a new
technique called Pairwise Ranking Prompting (PRP). Our results are the first in
the literature to achieve state-of-the-art ranking performance on standard
benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on
the Flan-UL2 model with 20B parameters outperforms the previous best approach
in the literature, which is based on the blackbox commercial GPT-4 that has 50x
(estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only
inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while
outperforming other existing solutions, such as InstructGPT which has 175B
parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose
several variants of PRP to improve efficiency and show that it is possible to
achieve competitive results even with linear complexity. We also discuss other
benefits of PRP, such as supporting both generation and scoring LLM APIs, as
well as being insensitive to input ordering.
64a22028a1a76b1d016e3363
Zhen Qin
claimed_verified
2024-06-18T08:29:51.947Z
64a2bd9245dde1d6ac7b4d81
/avatars/8a238a832bfcf14e7704a2152b254a0e.svg
Zhen Qin
user
paulqin
64a22028a1a76b1d016e3364
Rolf Jagerman
claimed_verified
2023-07-03T15:22:52.843Z
64341543a5aed21dd113bc30
https://cdn-avatars.huggingface.co/v1/production/uploads/64341543a5aed21dd113bc30/wFZmixmvhBd30iamJuoiM.jpeg
Rolf Jagerman
user
rjagerman
64a22028a1a76b1d016e3365
Kai Hui
64a22028a1a76b1d016e3366
Honglei Zhuang
64a22028a1a76b1d016e3367
Junru Wu
64a22028a1a76b1d016e3368
Jiaming Shen
64a22028a1a76b1d016e3369
Tianqi Liu
claimed_verified
2023-09-15T07:41:14.365Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
64a22028a1a76b1d016e336a
Jialu Liu
64a22028a1a76b1d016e336b
Donald Metzler
64a22028a1a76b1d016e336c
Xuanhui Wang
64a22028a1a76b1d016e336d
Michael Bendersky
Large Language Models are Effective Text Rankers with Pairwise Ranking
  Prompting
Ranking documents using Large Language Models (LLMs) by directly feeding the
query and candidate documents into the prompt is an interesting and practical
problem. However, there has been limited success so far, as researchers have
found it difficult to outperform fine-tuned baseline rankers on benchmark
datasets. We analyze pointwise and listwise ranking prompts used by existing
methods and argue that off-the-shelf LLMs do not fully understand these ranking
formulations, possibly due to the nature of how LLMs are trained. In this
paper, we propose to significantly reduce the burden on LLMs by using a new
technique called Pairwise Ranking Prompting (PRP). Our results are the first in
the literature to achieve state-of-the-art ranking performance on standard
benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on
the Flan-UL2 model with 20B parameters outperforms the previous best approach
in the literature, which is based on the blackbox commercial GPT-4 that has 50x
(estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only
inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while
outperforming other existing solutions, such as InstructGPT which has 175B
parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose
several variants of PRP to improve efficiency and show that it is possible to
achieve competitive results even with linear complexity. We also discuss other
benefits of PRP, such as supporting both generation and scoring LLM APIs, as
well as being insensitive to input ordering.
A pairwise ranking prompting technique for document ranking using large language models achieves state-of-the-art performance with moderate-sized, open-sourced models compared to larger, commercial models.
Large Language Models Pairwise Ranking Prompting pointwise ranking listwise ranking prompt query candidate documents benchmark datasets fine-tuned rankers TREC-DL2020 Flan-UL2 GPT-4 NDCG@1 NDCG@5 NDCG@10 InstructGPT document ranking generation scoring LLM APIs