title: Understand What LLM Needs: Dual Preference Alignment for
    Retrieval-Augmented Generation
summary: Retrieval-augmented generation (RAG) has demonstrated effectiveness in
  mitigating the hallucination problem of large language models (LLMs). However,
  the difficulty of aligning the retriever with the diverse LLMs' knowledge
  preferences inevitably poses an inevitable challenge in developing a reliable
  RAG system. To address this issue, we propose DPA-RAG, a universal framework
  designed to align diverse knowledge preferences within RAG systems.
  Specifically, we initially introduce a preference knowledge construction
  pipline and incorporate five novel query augmentation strategies to alleviate
  preference data scarcity. Based on preference data, DPA-RAG accomplishes both
  external and internal preference alignment: 1) It jointly integrate pair-wise,
  point-wise, and contrastive preference alignment abilities into the reranker,
  achieving external preference alignment among RAG components. 2) It further
  introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT),
  enabling LLMs to implicitly capture knowledge aligned with their reasoning
  preferences, achieving LLMs' internal alignment. Experimental results across
  four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all
  baselines and seamlessly integrates both black-box and open-sourced LLM
  readers. Further qualitative analysis and discussions also provide empirical
  guidance for achieving reliable RAG systems. Our code is publicly available at
  https://github.com/dongguanting/DPA-RAG.
paper_authors._id: 667ee6f166aad3ce8aa0184f
paper_authors.name: Guanting Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:56:28.753Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 667ee6f166aad3ce8aa01850
paper_authors.name: Yutao Zhu
paper_authors._id: 667ee6f166aad3ce8aa01851
paper_authors.name: Chenghao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:05:08.971Z
paper_authors._id: 6710ac3fb4ee4920580a5f0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6710ac3fb4ee4920580a5f0e/OhQQFlZmkmLQpMYqKCGP6.jpeg
paper_authors.fullname: Chenghao Zhang
paper_authors.type: user
paper_authors.user: SnowNation
paper_authors._id: 667ee6f166aad3ce8aa01852
paper_authors.name: Zechen Wang
paper_authors._id: 667ee6f166aad3ce8aa01853
paper_authors.name: Zhicheng Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T06:58:49.853Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_authors._id: 667ee6f166aad3ce8aa01854
paper_authors.name: Ji-Rong Wen
paper_title: Understand What LLM Needs: Dual Preference Alignment for
    Retrieval-Augmented Generation
paper_summary: Retrieval-augmented generation (RAG) has demonstrated effectiveness in
  mitigating the hallucination problem of large language models (LLMs). However,
  the difficulty of aligning the retriever with the diverse LLMs' knowledge
  preferences inevitably poses an inevitable challenge in developing a reliable
  RAG system. To address this issue, we propose DPA-RAG, a universal framework
  designed to align diverse knowledge preferences within RAG systems.
  Specifically, we initially introduce a preference knowledge construction
  pipline and incorporate five novel query augmentation strategies to alleviate
  preference data scarcity. Based on preference data, DPA-RAG accomplishes both
  external and internal preference alignment: 1) It jointly integrate pair-wise,
  point-wise, and contrastive preference alignment abilities into the reranker,
  achieving external preference alignment among RAG components. 2) It further
  introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT),
  enabling LLMs to implicitly capture knowledge aligned with their reasoning
  preferences, achieving LLMs' internal alignment. Experimental results across
  four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all
  baselines and seamlessly integrates both black-box and open-sourced LLM
  readers. Further qualitative analysis and discussions also provide empirical
  guidance for achieving reliable RAG systems. Our code is publicly available at
  https://github.com/dongguanting/DPA-RAG.
paper_ai_summary: DPA-RAG mitigates the mismatch between retrievers and LLMs' knowledge preferences through preference alignment and query augmentation, enhancing RAG system reliability.
paper_ai_keywords: Retrieval-augmented generation LLMs hallucination proble preference knowledge construction query augmentation strategies pair-wise preference alignment point-wise preference alignment contrastive preference alignment reranker pre-aligned stage Supervised Fine-tuning knowledge-intensive QA datasets black-box LLM readers open-sourced LLM readers