Toward General Instruction-Following Alignment for Retrieval-Augmented
  Generation
Following natural instructions is crucial for the effective application of
Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in
Large Language Models (LLMs), research on assessing and improving
instruction-following (IF) alignment within the RAG domain remains limited. To
address this issue, we propose VIF-RAG, the first automated, scalable, and
verifiable synthetic pipeline for instruction-following alignment in RAG
systems. We start by manually crafting a minimal set of atomic instructions
(<100) and developing combination rules to synthesize and verify complex
instructions for a seed set. We then use supervised models for instruction
rewriting while simultaneously generating code to automate the verification of
instruction quality via a Python executor. Finally, we integrate these
instructions with extensive RAG and general data samples, scaling up to a
high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further
bridge the gap in instruction-following auto-evaluation for RAG systems, we
introduce FollowRAG Benchmark, which includes approximately 3K test samples,
covering 22 categories of general instruction constraints and four
knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG
can seamlessly integrate with different RAG benchmarks. Using FollowRAG and
eight widely-used IF and foundational abilities benchmarks for LLMs, we
demonstrate that VIF-RAG markedly enhances LLM performance across a broad range
of general instruction constraints while effectively leveraging its
capabilities in RAG scenarios. Further analysis offers practical insights for
achieving IF alignment in RAG systems. Our code and datasets are released at
https://FollowRAG.github.io.
670dcefa487a5e24b2768ecf
Guanting Dong
admin_assigned
2024-10-15T13:25:39.921Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
670dcefa487a5e24b2768ed0
Xiaoshuai Song
670dcefa487a5e24b2768ed1
Yutao Zhu
admin_assigned
2024-10-15T13:25:54.583Z
625e62452a7279d3c77b5c38
https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
Yutao Zhu
user
yutaozhu94
670dcefa487a5e24b2768ed2
Runqi Qiao
claimed_verified
2024-12-02T08:51:26.391Z
6683a05e74fb1736a4b7c934
/avatars/35292253cf1882080be353c4b8c9ca3b.svg
QRQ
user
RichardQRQ
670dcefa487a5e24b2768ed3
Zhicheng Dou
admin_assigned
2024-10-15T13:26:10.877Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
670dcefa487a5e24b2768ed4
Ji-Rong Wen
admin_assigned
2024-10-15T13:26:16.633Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
Toward General Instruction-Following Alignment for Retrieval-Augmented
  Generation
Following natural instructions is crucial for the effective application of
Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in
Large Language Models (LLMs), research on assessing and improving
instruction-following (IF) alignment within the RAG domain remains limited. To
address this issue, we propose VIF-RAG, the first automated, scalable, and
verifiable synthetic pipeline for instruction-following alignment in RAG
systems. We start by manually crafting a minimal set of atomic instructions
(<100) and developing combination rules to synthesize and verify complex
instructions for a seed set. We then use supervised models for instruction
rewriting while simultaneously generating code to automate the verification of
instruction quality via a Python executor. Finally, we integrate these
instructions with extensive RAG and general data samples, scaling up to a
high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further
bridge the gap in instruction-following auto-evaluation for RAG systems, we
introduce FollowRAG Benchmark, which includes approximately 3K test samples,
covering 22 categories of general instruction constraints and four
knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG
can seamlessly integrate with different RAG benchmarks. Using FollowRAG and
eight widely-used IF and foundational abilities benchmarks for LLMs, we
demonstrate that VIF-RAG markedly enhances LLM performance across a broad range
of general instruction constraints while effectively leveraging its
capabilities in RAG scenarios. Further analysis offers practical insights for
achieving IF alignment in RAG systems. Our code and datasets are released at
https://FollowRAG.github.io.
VIF-RAG-QA and FollowRAG Benchmark enhance Instruction-Following alignment in RAG systems through automated instruction synthesis, verification, and evaluation.
Retrieval-Augmented Generation (RAG) Large Language Models (LLMs) instruction-following (IF) alignment VIF-RAG synthetic pipeline atomic instructions combination rules instruction rewriting Python executor VIF-RAG-QA dataset FollowRAG Benchmark instruction constraints knowledge-intensive QA datasets LLM performance