title: IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in
    Expert-Domain Information Retrieval
summary: We introduce IFIR, the first comprehensive benchmark designed to evaluate
  instruction-following information retrieval (IR) in expert domains. IFIR
  includes 2,426 high-quality examples and covers eight subsets across four
  specialized domains: finance, law, healthcare, and science literature. Each
  subset addresses one or more domain-specific retrieval tasks, replicating
  real-world scenarios where customized instructions are critical. IFIR enables a
  detailed analysis of instruction-following retrieval capabilities by
  incorporating instructions at different levels of complexity. We also propose a
  novel LLM-based evaluation method to provide a more precise and reliable
  assessment of model performance in following instructions. Through extensive
  experiments on 15 frontier retrieval models, including those based on LLMs, our
  results reveal that current models face significant challenges in effectively
  following complex, domain-specific instructions. We further provide in-depth
  analyses to highlight these limitations, offering valuable insights to guide
  future advancements in retriever development.
paper_authors._id: 67ca5d2783ac16a063a56241
paper_authors.name: Tingyu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:53:24.813Z
paper_authors._id: 64dc29d9b5d625e0e9a6ecb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
paper_authors.fullname: Tingyu Song
paper_authors.type: user
paper_authors.user: songtingyu
paper_authors._id: 67ca5d2783ac16a063a56242
paper_authors.name: Guo Gan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:09:52.646Z
paper_authors._id: 65dfeee3d16fb170031df293
paper_authors.avatarUrl: /avatars/05e6fe0e61d4bb87536554c782385dac.svg
paper_authors.fullname: gan
paper_authors.type: user
paper_authors.user: guo9
paper_authors._id: 67ca5d2783ac16a063a56243
paper_authors.name: Mingsheng Shang
paper_authors._id: 67ca5d2783ac16a063a56244
paper_authors.name: Yilun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:52:59.499Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_title: IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in
    Expert-Domain Information Retrieval
paper_summary: We introduce IFIR, the first comprehensive benchmark designed to evaluate
  instruction-following information retrieval (IR) in expert domains. IFIR
  includes 2,426 high-quality examples and covers eight subsets across four
  specialized domains: finance, law, healthcare, and science literature. Each
  subset addresses one or more domain-specific retrieval tasks, replicating
  real-world scenarios where customized instructions are critical. IFIR enables a
  detailed analysis of instruction-following retrieval capabilities by
  incorporating instructions at different levels of complexity. We also propose a
  novel LLM-based evaluation method to provide a more precise and reliable
  assessment of model performance in following instructions. Through extensive
  experiments on 15 frontier retrieval models, including those based on LLMs, our
  results reveal that current models face significant challenges in effectively
  following complex, domain-specific instructions. We further provide in-depth
  analyses to highlight these limitations, offering valuable insights to guide
  future advancements in retriever development.
paper_ai_summary: A benchmark and evaluation method for instruction-following information retrieval in expert domains reveal challenges for current models in handling complex, domain-specific instructions.
paper_ai_keywords: instruction-following information retrieval benchmark LLM-based evaluation method retriever development