Promptriever: Instruction-Trained Retrievers Can Be Prompted Like
  Language Models
Instruction-tuned language models (LM) are able to respond to imperative
commands, providing a more natural user interface compared to their base
counterparts. In this work, we present Promptriever, the first retrieval model
able to be prompted like an LM. To train Promptriever, we curate and release a
new instance-level instruction training set from MS MARCO, spanning nearly 500k
instances. Promptriever not only achieves strong performance on standard
retrieval tasks, but also follows instructions. We observe: (1) large gains
(reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR /
+3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical
choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR),
and (3) the ability to perform hyperparameter search via prompting to reliably
improve retrieval performance (+1.4 average increase on BEIR). Promptriever
demonstrates that retrieval models can be controlled with prompts on a
per-query basis, setting the stage for future work aligning LM prompting
techniques with information retrieval.
66ea2bb2c7dd4e003cd574ce
Orion Weller
claimed_verified
2024-12-19T18:05:49.426Z
6362d9712691058b19de1ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
Orion Weller
user
orionweller
66ea2bb2c7dd4e003cd574cf
Benjamin Van Durme
66ea2bb2c7dd4e003cd574d0
Dawn Lawrie
66ea2bb2c7dd4e003cd574d1
Ashwin Paranjape
66ea2bb2c7dd4e003cd574d2
Yuhao Zhang
66ea2bb2c7dd4e003cd574d3
Jack Hessel
Promptriever: Instruction-Trained Retrievers Can Be Prompted Like
  Language Models
Instruction-tuned language models (LM) are able to respond to imperative
commands, providing a more natural user interface compared to their base
counterparts. In this work, we present Promptriever, the first retrieval model
able to be prompted like an LM. To train Promptriever, we curate and release a
new instance-level instruction training set from MS MARCO, spanning nearly 500k
instances. Promptriever not only achieves strong performance on standard
retrieval tasks, but also follows instructions. We observe: (1) large gains
(reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR /
+3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical
choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR),
and (3) the ability to perform hyperparameter search via prompting to reliably
improve retrieval performance (+1.4 average increase on BEIR). Promptriever
demonstrates that retrieval models can be controlled with prompts on a
per-query basis, setting the stage for future work aligning LM prompting
techniques with information retrieval.
Promptriever, a retrieval model trained to respond to prompts, achieves state-of-the-art performance on instruction-following tasks and demonstrates improved robustness and hyperparameter optimization capacity.
instruction-tuned language models Promptriever MS MARCO FollowIR InstructIR BEIR hyperparameter search retrieval models prompting