Rank-without-GPT: Building GPT-Independent Listwise Rerankers on
  Open-Source Large Language Models
Listwise rerankers based on large language models (LLM) are the zero-shot
state-of-the-art. However, current works in this direction all depend on the
GPT models, making it a single point of failure in scientific reproducibility.
Moreover, it raises the concern that the current research findings only hold
for GPT models but not LLM in general. In this work, we lift this pre-condition
and build for the first time effective listwise rerankers without any form of
dependency on GPT. Our passage retrieval experiments show that our best list se
reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves
97% effectiveness of the ones built on GPT-4. Our results also show that the
existing training datasets, which were expressly constructed for pointwise
ranking, are insufficient for building such listwise rerankers. Instead,
high-quality listwise ranking data is required and crucial, calling for further
work on building human-annotated listwise data resources.
656fd36cb4ebaeb855d4bf37
Xinyu Zhang
claimed_verified
2023-12-07T08:30:07.275Z
611fb1cbfa8355ed0309de81
https://cdn-avatars.huggingface.co/v1/production/uploads/1665676377085-611fb1cbfa8355ed0309de81.jpeg
Xinyu ZHANG
user
crystina-z
656fd36cb4ebaeb855d4bf38
Sebastian Hofstätter
admin_assigned
2023-12-06T13:15:15.607Z
60506d8a5f211e59bdaff26b
https://cdn-avatars.huggingface.co/v1/production/uploads/1616085391299-60506d8a5f211e59bdaff26b.jpeg
Sebastian Hofstätter
user
sebastian-hofstaetter
656fd36cb4ebaeb855d4bf39
Patrick Lewis
656fd36cb4ebaeb855d4bf3a
Raphael Tang
656fd36cb4ebaeb855d4bf3b
Jimmy Lin
Rank-without-GPT: Building GPT-Independent Listwise Rerankers on
  Open-Source Large Language Models
Listwise rerankers based on large language models (LLM) are the zero-shot
state-of-the-art. However, current works in this direction all depend on the
GPT models, making it a single point of failure in scientific reproducibility.
Moreover, it raises the concern that the current research findings only hold
for GPT models but not LLM in general. In this work, we lift this pre-condition
and build for the first time effective listwise rerankers without any form of
dependency on GPT. Our passage retrieval experiments show that our best list se
reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves
97% effectiveness of the ones built on GPT-4. Our results also show that the
existing training datasets, which were expressly constructed for pointwise
ranking, are insufficient for building such listwise rerankers. Instead,
high-quality listwise ranking data is required and crucial, calling for further
work on building human-annotated listwise data resources.
The study develops effective listwise rerankers independent of GPT models, surpassing GPT-3.5 and achieving near-GPT-4 performance, and highlights the need for high-quality listwise ranking datasets.
listwise rerankers large language models LLM GPT models passage retrieval list se reranker pointwise ranking high-quality listwise ranking data human-annotated listwise data resources