title: Rank-without-GPT: Building GPT-Independent Listwise Rerankers on
    Open-Source Large Language Models
summary: Listwise rerankers based on large language models (LLM) are the zero-shot
  state-of-the-art. However, current works in this direction all depend on the
  GPT models, making it a single point of failure in scientific reproducibility.
  Moreover, it raises the concern that the current research findings only hold
  for GPT models but not LLM in general. In this work, we lift this pre-condition
  and build for the first time effective listwise rerankers without any form of
  dependency on GPT. Our passage retrieval experiments show that our best list se
  reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves
  97% effectiveness of the ones built on GPT-4. Our results also show that the
  existing training datasets, which were expressly constructed for pointwise
  ranking, are insufficient for building such listwise rerankers. Instead,
  high-quality listwise ranking data is required and crucial, calling for further
  work on building human-annotated listwise data resources.
paper_authors._id: 656fd36cb4ebaeb855d4bf37
paper_authors.name: Xinyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:30:07.275Z
paper_authors._id: 611fb1cbfa8355ed0309de81
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665676377085-611fb1cbfa8355ed0309de81.jpeg
paper_authors.fullname: Xinyu ZHANG
paper_authors.type: user
paper_authors.user: crystina-z
paper_authors._id: 656fd36cb4ebaeb855d4bf38
paper_authors.name: Sebastian Hofstätter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:15:15.607Z
paper_authors._id: 60506d8a5f211e59bdaff26b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616085391299-60506d8a5f211e59bdaff26b.jpeg
paper_authors.fullname: Sebastian Hofstätter
paper_authors.type: user
paper_authors.user: sebastian-hofstaetter
paper_authors._id: 656fd36cb4ebaeb855d4bf39
paper_authors.name: Patrick Lewis
paper_authors._id: 656fd36cb4ebaeb855d4bf3a
paper_authors.name: Raphael Tang
paper_authors._id: 656fd36cb4ebaeb855d4bf3b
paper_authors.name: Jimmy Lin
paper_title: Rank-without-GPT: Building GPT-Independent Listwise Rerankers on
    Open-Source Large Language Models
paper_summary: Listwise rerankers based on large language models (LLM) are the zero-shot
  state-of-the-art. However, current works in this direction all depend on the
  GPT models, making it a single point of failure in scientific reproducibility.
  Moreover, it raises the concern that the current research findings only hold
  for GPT models but not LLM in general. In this work, we lift this pre-condition
  and build for the first time effective listwise rerankers without any form of
  dependency on GPT. Our passage retrieval experiments show that our best list se
  reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves
  97% effectiveness of the ones built on GPT-4. Our results also show that the
  existing training datasets, which were expressly constructed for pointwise
  ranking, are insufficient for building such listwise rerankers. Instead,
  high-quality listwise ranking data is required and crucial, calling for further
  work on building human-annotated listwise data resources.
paper_ai_summary: The study develops effective listwise rerankers independent of GPT models, surpassing GPT-3.5 and achieving near-GPT-4 performance, and highlights the need for high-quality listwise ranking datasets.
paper_ai_keywords: listwise rerankers large language models LLM GPT models passage retrieval list se reranker pointwise ranking high-quality listwise ranking data human-annotated listwise data resources