title: RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented
    Generation for Preference Alignment
summary: Despite the significant progress made by existing retrieval augmented
  language models (RALMs) in providing trustworthy responses and grounding in
  reliable sources, they often overlook effective alignment with human
  preferences. In the alignment process, reward models (RMs) act as a crucial
  proxy for human values to guide optimization. However, it remains unclear how
  to evaluate and select a reliable RM for preference alignment in RALMs. To this
  end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG
  settings. First, we design four crucial and challenging RAG-specific scenarios
  to assess RMs, including multi-hop reasoning, fine-grained citation,
  appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG
  subsets, six retrievers, and 24 RALMs to increase the diversity of data
  sources. Finally, we adopt an LLM-as-a-judge approach to improve preference
  annotation efficiency and effectiveness, exhibiting a strong correlation with
  human annotations. Based on the RAG-RewardBench, we conduct a comprehensive
  evaluation of 45 RMs and uncover their limitations in RAG scenarios.
  Additionally, we also reveal that existing trained RALMs show almost no
  improvement in preference alignment, highlighting the need for a shift towards
  preference-aligned training.We release our benchmark and code publicly at
  https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
paper_authors._id: 67638ce2995c144ae40aaf21
paper_authors.name: Zhuoran Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:08:30.400Z
paper_authors._id: 643379416c6ecd58798421b3
paper_authors.avatarUrl: /avatars/831db7eab2663abc33b176cf386b02f2.svg
paper_authors.fullname: Zhuoran Jin
paper_authors.type: user
paper_authors.user: jinzhuoran
paper_authors._id: 67638ce2995c144ae40aaf22
paper_authors.name: Hongbang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:08:37.222Z
paper_authors._id: 6538d3089d66a6c304ee2f4b
paper_authors.avatarUrl: /avatars/6e93ea6c937128765e5323fb4a5d1cea.svg
paper_authors.fullname: HongbangYuan
paper_authors.type: user
paper_authors.user: HongbangYuan
paper_authors._id: 67638ce2995c144ae40aaf23
paper_authors.name: Tianyi Men
paper_authors._id: 67638ce2995c144ae40aaf24
paper_authors.name: Pengfei Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:08:59.825Z
paper_authors._id: 6552d416b2d47c6675c580df
paper_authors.avatarUrl: /avatars/b3bb5332e716a968ca8b8d92e5766cf2.svg
paper_authors.fullname: caopengfei
paper_authors.type: user
paper_authors.user: pengfei0626
paper_authors._id: 67638ce2995c144ae40aaf25
paper_authors.name: Yubo Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:09:19.607Z
paper_authors._id: 64423a910f373bdb0463bf25
paper_authors.avatarUrl: /avatars/6b41c4eb817836d50d0072009f124f29.svg
paper_authors.fullname: Yubo Chen
paper_authors.type: user
paper_authors.user: ybch14
paper_authors._id: 67638ce2995c144ae40aaf26
paper_authors.name: Kang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:09:45.059Z
paper_authors._id: 62d18d7d5f924243758dd23c
paper_authors.avatarUrl: /avatars/d9d575558f10be661fe856236f4b72e3.svg
paper_authors.fullname: Kang Liu
paper_authors.type: user
paper_authors.user: cruyffio
paper_authors._id: 67638ce2995c144ae40aaf27
paper_authors.name: Jun Zhao
paper_title: RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented
    Generation for Preference Alignment
paper_summary: Despite the significant progress made by existing retrieval augmented
  language models (RALMs) in providing trustworthy responses and grounding in
  reliable sources, they often overlook effective alignment with human
  preferences. In the alignment process, reward models (RMs) act as a crucial
  proxy for human values to guide optimization. However, it remains unclear how
  to evaluate and select a reliable RM for preference alignment in RALMs. To this
  end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG
  settings. First, we design four crucial and challenging RAG-specific scenarios
  to assess RMs, including multi-hop reasoning, fine-grained citation,
  appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG
  subsets, six retrievers, and 24 RALMs to increase the diversity of data
  sources. Finally, we adopt an LLM-as-a-judge approach to improve preference
  annotation efficiency and effectiveness, exhibiting a strong correlation with
  human annotations. Based on the RAG-RewardBench, we conduct a comprehensive
  evaluation of 45 RMs and uncover their limitations in RAG scenarios.
  Additionally, we also reveal that existing trained RALMs show almost no
  improvement in preference alignment, highlighting the need for a shift towards
  preference-aligned training.We release our benchmark and code publicly at
  https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
paper_ai_summary: A benchmark named RAG-RewardBench evaluates and selects reward models for preference alignment in retrieval augmented language models, uncovering their limitations and highlighting the need for preference-aligned training.
paper_ai_keywords: retrieval augmented language models RALMs reward models RMs RAG-RewardBench multi-hop reasoning fine-grained citation appropriate abstain conflict robustness LLM-as-a-judge