title: Toward General Instruction-Following Alignment for Retrieval-Augmented
    Generation
summary: Following natural instructions is crucial for the effective application of
  Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in
  Large Language Models (LLMs), research on assessing and improving
  instruction-following (IF) alignment within the RAG domain remains limited. To
  address this issue, we propose VIF-RAG, the first automated, scalable, and
  verifiable synthetic pipeline for instruction-following alignment in RAG
  systems. We start by manually crafting a minimal set of atomic instructions
  (<100) and developing combination rules to synthesize and verify complex
  instructions for a seed set. We then use supervised models for instruction
  rewriting while simultaneously generating code to automate the verification of
  instruction quality via a Python executor. Finally, we integrate these
  instructions with extensive RAG and general data samples, scaling up to a
  high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further
  bridge the gap in instruction-following auto-evaluation for RAG systems, we
  introduce FollowRAG Benchmark, which includes approximately 3K test samples,
  covering 22 categories of general instruction constraints and four
  knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG
  can seamlessly integrate with different RAG benchmarks. Using FollowRAG and
  eight widely-used IF and foundational abilities benchmarks for LLMs, we
  demonstrate that VIF-RAG markedly enhances LLM performance across a broad range
  of general instruction constraints while effectively leveraging its
  capabilities in RAG scenarios. Further analysis offers practical insights for
  achieving IF alignment in RAG systems. Our code and datasets are released at
  https://FollowRAG.github.io.
paper_authors._id: 670dcefa487a5e24b2768ecf
paper_authors.name: Guanting Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:25:39.921Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 670dcefa487a5e24b2768ed0
paper_authors.name: Xiaoshuai Song
paper_authors._id: 670dcefa487a5e24b2768ed1
paper_authors.name: Yutao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:25:54.583Z
paper_authors._id: 625e62452a7279d3c77b5c38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
paper_authors.fullname: Yutao Zhu
paper_authors.type: user
paper_authors.user: yutaozhu94
paper_authors._id: 670dcefa487a5e24b2768ed2
paper_authors.name: Runqi Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:26.391Z
paper_authors._id: 6683a05e74fb1736a4b7c934
paper_authors.avatarUrl: /avatars/35292253cf1882080be353c4b8c9ca3b.svg
paper_authors.fullname: QRQ
paper_authors.type: user
paper_authors.user: RichardQRQ
paper_authors._id: 670dcefa487a5e24b2768ed3
paper_authors.name: Zhicheng Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:26:10.877Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_authors._id: 670dcefa487a5e24b2768ed4
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:26:16.633Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: Toward General Instruction-Following Alignment for Retrieval-Augmented
    Generation
paper_summary: Following natural instructions is crucial for the effective application of
  Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in
  Large Language Models (LLMs), research on assessing and improving
  instruction-following (IF) alignment within the RAG domain remains limited. To
  address this issue, we propose VIF-RAG, the first automated, scalable, and
  verifiable synthetic pipeline for instruction-following alignment in RAG
  systems. We start by manually crafting a minimal set of atomic instructions
  (<100) and developing combination rules to synthesize and verify complex
  instructions for a seed set. We then use supervised models for instruction
  rewriting while simultaneously generating code to automate the verification of
  instruction quality via a Python executor. Finally, we integrate these
  instructions with extensive RAG and general data samples, scaling up to a
  high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further
  bridge the gap in instruction-following auto-evaluation for RAG systems, we
  introduce FollowRAG Benchmark, which includes approximately 3K test samples,
  covering 22 categories of general instruction constraints and four
  knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG
  can seamlessly integrate with different RAG benchmarks. Using FollowRAG and
  eight widely-used IF and foundational abilities benchmarks for LLMs, we
  demonstrate that VIF-RAG markedly enhances LLM performance across a broad range
  of general instruction constraints while effectively leveraging its
  capabilities in RAG scenarios. Further analysis offers practical insights for
  achieving IF alignment in RAG systems. Our code and datasets are released at
  https://FollowRAG.github.io.
paper_ai_summary: VIF-RAG-QA and FollowRAG Benchmark enhance Instruction-Following alignment in RAG systems through automated instruction synthesis, verification, and evaluation.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) Large Language Models (LLMs) instruction-following (IF) alignment VIF-RAG synthetic pipeline atomic instructions combination rules instruction rewriting Python executor VIF-RAG-QA dataset FollowRAG Benchmark instruction constraints knowledge-intensive QA datasets LLM performance