title: FollowIR: Evaluating and Teaching Information Retrieval Models to Follow
    Instructions
summary: Modern Large Language Models (LLMs) are capable of following long and complex
  instructions that enable a diverse amount of user tasks. However, despite
  Information Retrieval (IR) models using LLMs as the backbone of their
  architectures, nearly all of them still only take queries as input, with no
  instructions. For the handful of recent models that do take instructions, it's
  unclear how they use them. We introduce our dataset FollowIR, which contains a
  rigorous instruction evaluation benchmark as well as a training set for helping
  IR models learn to better follow real-world instructions. FollowIR builds off
  the long history of the TREC conferences: as TREC provides human annotators
  with instructions (also known as narratives) to determine document relevance,
  so should IR models be able to understand and decide relevance based on these
  detailed instructions. Our evaluation benchmark starts with three deeply judged
  TREC collections and alters the annotator instructions, re-annotating relevant
  documents. Through this process, we can measure how well IR models follow
  instructions, through a new pairwise evaluation framework. Our results indicate
  that existing retrieval models fail to correctly use instructions, using them
  for basic keywords and struggling to understand long-form information. However,
  we show that it is possible for IR models to learn to follow complex
  instructions: our new FollowIR-7B model has significant improvements (over 13%)
  after fine-tuning on our training set.
paper_authors._id: 6600fdafd157381f16f4297c
paper_authors.name: Orion Weller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:02:38.518Z
paper_authors._id: 6362d9712691058b19de1ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
paper_authors.fullname: Orion Weller
paper_authors.type: user
paper_authors.user: orionweller
paper_authors._id: 6600fdafd157381f16f4297d
paper_authors.name: Benjamin Chang
paper_authors._id: 6600fdafd157381f16f4297e
paper_authors.name: Sean MacAvaney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:03:01.317Z
paper_authors._id: 61dc410f7813d319d7387635
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666975288423-61dc410f7813d319d7387635.jpeg
paper_authors.fullname: Sean MacAvaney
paper_authors.type: user
paper_authors.user: macavaney
paper_authors._id: 6600fdafd157381f16f4297f
paper_authors.name: Kyle Lo
paper_authors._id: 6600fdafd157381f16f42980
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:03:12.243Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_authors._id: 6600fdafd157381f16f42981
paper_authors.name: Benjamin Van Durme
paper_authors._id: 6600fdafd157381f16f42982
paper_authors.name: Dawn Lawrie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:03:23.291Z
paper_authors._id: 63a0c1ae3c8841cfe2cd3c90
paper_authors.avatarUrl: /avatars/b2495c05e0838ad117bf607ccddbb3e4.svg
paper_authors.fullname: Dawn Lawrie
paper_authors.type: user
paper_authors.user: dlawrie
paper_authors._id: 6600fdafd157381f16f42983
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:03:29.380Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_title: FollowIR: Evaluating and Teaching Information Retrieval Models to Follow
    Instructions
paper_summary: Modern Large Language Models (LLMs) are capable of following long and complex
  instructions that enable a diverse amount of user tasks. However, despite
  Information Retrieval (IR) models using LLMs as the backbone of their
  architectures, nearly all of them still only take queries as input, with no
  instructions. For the handful of recent models that do take instructions, it's
  unclear how they use them. We introduce our dataset FollowIR, which contains a
  rigorous instruction evaluation benchmark as well as a training set for helping
  IR models learn to better follow real-world instructions. FollowIR builds off
  the long history of the TREC conferences: as TREC provides human annotators
  with instructions (also known as narratives) to determine document relevance,
  so should IR models be able to understand and decide relevance based on these
  detailed instructions. Our evaluation benchmark starts with three deeply judged
  TREC collections and alters the annotator instructions, re-annotating relevant
  documents. Through this process, we can measure how well IR models follow
  instructions, through a new pairwise evaluation framework. Our results indicate
  that existing retrieval models fail to correctly use instructions, using them
  for basic keywords and struggling to understand long-form information. However,
  we show that it is possible for IR models to learn to follow complex
  instructions: our new FollowIR-7B model has significant improvements (over 13%)
  after fine-tuning on our training set.
paper_ai_keywords: Large Language Models (LLMs) Information Retrieval (IR) FollowIR TREC conferences narratives pairwise evaluation framework FollowIR-7B