Understand What LLM Needs: Dual Preference Alignment for
  Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) has demonstrated effectiveness in
mitigating the hallucination problem of large language models (LLMs). However,
the difficulty of aligning the retriever with the diverse LLMs' knowledge
preferences inevitably poses an inevitable challenge in developing a reliable
RAG system. To address this issue, we propose DPA-RAG, a universal framework
designed to align diverse knowledge preferences within RAG systems.
Specifically, we initially introduce a preference knowledge construction
pipline and incorporate five novel query augmentation strategies to alleviate
preference data scarcity. Based on preference data, DPA-RAG accomplishes both
external and internal preference alignment: 1) It jointly integrate pair-wise,
point-wise, and contrastive preference alignment abilities into the reranker,
achieving external preference alignment among RAG components. 2) It further
introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT),
enabling LLMs to implicitly capture knowledge aligned with their reasoning
preferences, achieving LLMs' internal alignment. Experimental results across
four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all
baselines and seamlessly integrates both black-box and open-sourced LLM
readers. Further qualitative analysis and discussions also provide empirical
guidance for achieving reliable RAG systems. Our code is publicly available at
https://github.com/dongguanting/DPA-RAG.
667ee6f166aad3ce8aa0184f
Guanting Dong
claimed_verified
2024-06-30T20:56:28.753Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
667ee6f166aad3ce8aa01850
Yutao Zhu
667ee6f166aad3ce8aa01851
Chenghao Zhang
claimed_verified
2024-12-17T08:05:08.971Z
6710ac3fb4ee4920580a5f0e
https://cdn-avatars.huggingface.co/v1/production/uploads/6710ac3fb4ee4920580a5f0e/OhQQFlZmkmLQpMYqKCGP6.jpeg
Chenghao Zhang
user
SnowNation
667ee6f166aad3ce8aa01852
Zechen Wang
667ee6f166aad3ce8aa01853
Zhicheng Dou
claimed_verified
2024-09-23T06:58:49.853Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
667ee6f166aad3ce8aa01854
Ji-Rong Wen
Understand What LLM Needs: Dual Preference Alignment for
  Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) has demonstrated effectiveness in
mitigating the hallucination problem of large language models (LLMs). However,
the difficulty of aligning the retriever with the diverse LLMs' knowledge
preferences inevitably poses an inevitable challenge in developing a reliable
RAG system. To address this issue, we propose DPA-RAG, a universal framework
designed to align diverse knowledge preferences within RAG systems.
Specifically, we initially introduce a preference knowledge construction
pipline and incorporate five novel query augmentation strategies to alleviate
preference data scarcity. Based on preference data, DPA-RAG accomplishes both
external and internal preference alignment: 1) It jointly integrate pair-wise,
point-wise, and contrastive preference alignment abilities into the reranker,
achieving external preference alignment among RAG components. 2) It further
introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT),
enabling LLMs to implicitly capture knowledge aligned with their reasoning
preferences, achieving LLMs' internal alignment. Experimental results across
four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all
baselines and seamlessly integrates both black-box and open-sourced LLM
readers. Further qualitative analysis and discussions also provide empirical
guidance for achieving reliable RAG systems. Our code is publicly available at
https://github.com/dongguanting/DPA-RAG.
DPA-RAG mitigates the mismatch between retrievers and LLMs' knowledge preferences through preference alignment and query augmentation, enhancing RAG system reliability.
Retrieval-augmented generation LLMs hallucination proble preference knowledge construction query augmentation strategies pair-wise preference alignment point-wise preference alignment contrastive preference alignment reranker pre-aligned stage Supervised Fine-tuning knowledge-intensive QA datasets black-box LLM readers open-sourced LLM readers