title: Large Language Models are Effective Text Rankers with Pairwise Ranking
    Prompting
summary: Ranking documents using Large Language Models (LLMs) by directly feeding the
  query and candidate documents into the prompt is an interesting and practical
  problem. However, there has been limited success so far, as researchers have
  found it difficult to outperform fine-tuned baseline rankers on benchmark
  datasets. We analyze pointwise and listwise ranking prompts used by existing
  methods and argue that off-the-shelf LLMs do not fully understand these ranking
  formulations, possibly due to the nature of how LLMs are trained. In this
  paper, we propose to significantly reduce the burden on LLMs by using a new
  technique called Pairwise Ranking Prompting (PRP). Our results are the first in
  the literature to achieve state-of-the-art ranking performance on standard
  benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on
  the Flan-UL2 model with 20B parameters outperforms the previous best approach
  in the literature, which is based on the blackbox commercial GPT-4 that has 50x
  (estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only
  inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while
  outperforming other existing solutions, such as InstructGPT which has 175B
  parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose
  several variants of PRP to improve efficiency and show that it is possible to
  achieve competitive results even with linear complexity. We also discuss other
  benefits of PRP, such as supporting both generation and scoring LLM APIs, as
  well as being insensitive to input ordering.
paper_authors._id: 64a22028a1a76b1d016e3363
paper_authors.name: Zhen Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T08:29:51.947Z
paper_authors._id: 64a2bd9245dde1d6ac7b4d81
paper_authors.avatarUrl: /avatars/8a238a832bfcf14e7704a2152b254a0e.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: paulqin
paper_authors._id: 64a22028a1a76b1d016e3364
paper_authors.name: Rolf Jagerman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T15:22:52.843Z
paper_authors._id: 64341543a5aed21dd113bc30
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64341543a5aed21dd113bc30/wFZmixmvhBd30iamJuoiM.jpeg
paper_authors.fullname: Rolf Jagerman
paper_authors.type: user
paper_authors.user: rjagerman
paper_authors._id: 64a22028a1a76b1d016e3365
paper_authors.name: Kai Hui
paper_authors._id: 64a22028a1a76b1d016e3366
paper_authors.name: Honglei Zhuang
paper_authors._id: 64a22028a1a76b1d016e3367
paper_authors.name: Junru Wu
paper_authors._id: 64a22028a1a76b1d016e3368
paper_authors.name: Jiaming Shen
paper_authors._id: 64a22028a1a76b1d016e3369
paper_authors.name: Tianqi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-15T07:41:14.365Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 64a22028a1a76b1d016e336a
paper_authors.name: Jialu Liu
paper_authors._id: 64a22028a1a76b1d016e336b
paper_authors.name: Donald Metzler
paper_authors._id: 64a22028a1a76b1d016e336c
paper_authors.name: Xuanhui Wang
paper_authors._id: 64a22028a1a76b1d016e336d
paper_authors.name: Michael Bendersky
paper_title: Large Language Models are Effective Text Rankers with Pairwise Ranking
    Prompting
paper_summary: Ranking documents using Large Language Models (LLMs) by directly feeding the
  query and candidate documents into the prompt is an interesting and practical
  problem. However, there has been limited success so far, as researchers have
  found it difficult to outperform fine-tuned baseline rankers on benchmark
  datasets. We analyze pointwise and listwise ranking prompts used by existing
  methods and argue that off-the-shelf LLMs do not fully understand these ranking
  formulations, possibly due to the nature of how LLMs are trained. In this
  paper, we propose to significantly reduce the burden on LLMs by using a new
  technique called Pairwise Ranking Prompting (PRP). Our results are the first in
  the literature to achieve state-of-the-art ranking performance on standard
  benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on
  the Flan-UL2 model with 20B parameters outperforms the previous best approach
  in the literature, which is based on the blackbox commercial GPT-4 that has 50x
  (estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only
  inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while
  outperforming other existing solutions, such as InstructGPT which has 175B
  parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose
  several variants of PRP to improve efficiency and show that it is possible to
  achieve competitive results even with linear complexity. We also discuss other
  benefits of PRP, such as supporting both generation and scoring LLM APIs, as
  well as being insensitive to input ordering.
paper_ai_keywords: Pairwise Ranking Prompting (PRP) TREC-DL2020 Flan-UL2 NDCG@1 TREC-DL2019 NDCG@5 NDCG@10 InstructGPT generation scoring LLM APIs