title: Agentic Reward Modeling: Integrating Human Preferences with Verifiable
    Correctness Signals for Reliable Reward Systems
summary: Reward models (RMs) are crucial for the training and inference-time scaling
  up of large language models (LLMs). However, existing reward models primarily
  focus on human preferences, neglecting verifiable correctness signals which
  have shown strong potential in training LLMs. In this paper, we propose agentic
  reward modeling, a reward system that combines reward models with verifiable
  correctness signals from different aspects to provide reliable rewards. We
  empirically implement a reward agent, named RewardAgent, that combines human
  preference rewards with two verifiable signals: factuality and instruction
  following, to provide more reliable rewards. We conduct comprehensive
  experiments on existing reward model benchmarks and inference time best-of-n
  searches on real-world downstream tasks. RewardAgent significantly outperforms
  vanilla reward models, demonstrating its effectiveness. We further construct
  training preference pairs using RewardAgent and train an LLM with the DPO
  objective, achieving superior performance on various NLP benchmarks compared to
  conventional reward models. Our codes are publicly released to facilitate
  further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).
paper_authors._id: 67bfcb774d22a9379b29334c
paper_authors.name: Hao Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:18:21.224Z
paper_authors._id: 625a5446f1063e7085d5178a
paper_authors.avatarUrl: /avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: Wesleythu
paper_authors._id: 67bfcb774d22a9379b29334d
paper_authors.name: Yunjia Qi
paper_authors._id: 67bfcb774d22a9379b29334e
paper_authors.name: Xiaozhi Wang
paper_authors._id: 67bfcb774d22a9379b29334f
paper_authors.name: Zijun Yao
paper_authors._id: 67bfcb774d22a9379b293350
paper_authors.name: Bin Xu
paper_authors._id: 67bfcb774d22a9379b293351
paper_authors.name: Lei Hou
paper_authors._id: 67bfcb774d22a9379b293352
paper_authors.name: Juanzi Li
paper_title: Agentic Reward Modeling: Integrating Human Preferences with Verifiable
    Correctness Signals for Reliable Reward Systems
paper_summary: Reward models (RMs) are crucial for the training and inference-time scaling
  up of large language models (LLMs). However, existing reward models primarily
  focus on human preferences, neglecting verifiable correctness signals which
  have shown strong potential in training LLMs. In this paper, we propose agentic
  reward modeling, a reward system that combines reward models with verifiable
  correctness signals from different aspects to provide reliable rewards. We
  empirically implement a reward agent, named RewardAgent, that combines human
  preference rewards with two verifiable signals: factuality and instruction
  following, to provide more reliable rewards. We conduct comprehensive
  experiments on existing reward model benchmarks and inference time best-of-n
  searches on real-world downstream tasks. RewardAgent significantly outperforms
  vanilla reward models, demonstrating its effectiveness. We further construct
  training preference pairs using RewardAgent and train an LLM with the DPO
  objective, achieving superior performance on various NLP benchmarks compared to
  conventional reward models. Our codes are publicly released to facilitate
  further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).
paper_ai_summary: An agentic reward system that combines human preferences with verifiable correctness signals, demonstrated through RewardAgent, enhances reward reliability and improves large language model performance.
paper_ai_keywords: reward models large language models verifiable correctness signals agentic reward modeling RewardAgent factuality instruction following DPO objective