FollowIR: Evaluating and Teaching Information Retrieval Models to Follow
  Instructions
Modern Large Language Models (LLMs) are capable of following long and complex
instructions that enable a diverse amount of user tasks. However, despite
Information Retrieval (IR) models using LLMs as the backbone of their
architectures, nearly all of them still only take queries as input, with no
instructions. For the handful of recent models that do take instructions, it's
unclear how they use them. We introduce our dataset FollowIR, which contains a
rigorous instruction evaluation benchmark as well as a training set for helping
IR models learn to better follow real-world instructions. FollowIR builds off
the long history of the TREC conferences: as TREC provides human annotators
with instructions (also known as narratives) to determine document relevance,
so should IR models be able to understand and decide relevance based on these
detailed instructions. Our evaluation benchmark starts with three deeply judged
TREC collections and alters the annotator instructions, re-annotating relevant
documents. Through this process, we can measure how well IR models follow
instructions, through a new pairwise evaluation framework. Our results indicate
that existing retrieval models fail to correctly use instructions, using them
for basic keywords and struggling to understand long-form information. However,
we show that it is possible for IR models to learn to follow complex
instructions: our new FollowIR-7B model has significant improvements (over 13%)
after fine-tuning on our training set.
6600fdafd157381f16f4297c
Orion Weller
admin_assigned
2024-03-25T11:02:38.518Z
6362d9712691058b19de1ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
Orion Weller
user
orionweller
6600fdafd157381f16f4297d
Benjamin Chang
6600fdafd157381f16f4297e
Sean MacAvaney
admin_assigned
2024-03-25T11:03:01.317Z
61dc410f7813d319d7387635
https://cdn-avatars.huggingface.co/v1/production/uploads/1666975288423-61dc410f7813d319d7387635.jpeg
Sean MacAvaney
user
macavaney
6600fdafd157381f16f4297f
Kyle Lo
6600fdafd157381f16f42980
Arman Cohan
admin_assigned
2024-03-25T11:03:12.243Z
5f5ba21188f57f65f951f255
https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
Arman Cohan
user
armanc
6600fdafd157381f16f42981
Benjamin Van Durme
6600fdafd157381f16f42982
Dawn Lawrie
admin_assigned
2024-03-25T11:03:23.291Z
63a0c1ae3c8841cfe2cd3c90
/avatars/b2495c05e0838ad117bf607ccddbb3e4.svg
Dawn Lawrie
user
dlawrie
6600fdafd157381f16f42983
Luca Soldaini
admin_assigned
2024-03-25T11:03:29.380Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow
  Instructions
Modern Large Language Models (LLMs) are capable of following long and complex
instructions that enable a diverse amount of user tasks. However, despite
Information Retrieval (IR) models using LLMs as the backbone of their
architectures, nearly all of them still only take queries as input, with no
instructions. For the handful of recent models that do take instructions, it's
unclear how they use them. We introduce our dataset FollowIR, which contains a
rigorous instruction evaluation benchmark as well as a training set for helping
IR models learn to better follow real-world instructions. FollowIR builds off
the long history of the TREC conferences: as TREC provides human annotators
with instructions (also known as narratives) to determine document relevance,
so should IR models be able to understand and decide relevance based on these
detailed instructions. Our evaluation benchmark starts with three deeply judged
TREC collections and alters the annotator instructions, re-annotating relevant
documents. Through this process, we can measure how well IR models follow
instructions, through a new pairwise evaluation framework. Our results indicate
that existing retrieval models fail to correctly use instructions, using them
for basic keywords and struggling to understand long-form information. However,
we show that it is possible for IR models to learn to follow complex
instructions: our new FollowIR-7B model has significant improvements (over 13%)
after fine-tuning on our training set.
FollowIR dataset and evaluation framework demonstrate that large language models can be fine-tuned to better follow detailed instructions in information retrieval, improving performance over baseline models.
Large Language Models FollowIR Information Retrieval TREC pairwise evaluation framework instruction evaluation benchmark document relevance fine-tuning