title: Reward-Robust RLHF in LLMs
summary: As Large Language Models (LLMs) continue to progress toward more advanced
  forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
  increasingly seen as a key pathway toward achieving Artificial General
  Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
  alignment methods introduces significant challenges due to the inherent
  instability and imperfections of Reward Models (RMs), which can lead to
  critical issues such as reward hacking and misalignment with human intentions.
  In this paper, we introduce a reward-robust RLHF framework aimed at addressing
  these fundamental challenges, paving the way for more reliable and resilient
  learning in LLMs. Our approach introduces a novel optimization objective that
  carefully balances performance and robustness by incorporating Bayesian Reward
  Model Ensembles (BRME) to model the uncertainty set of reward functions. This
  allows the framework to integrate both nominal performance and minimum reward
  signals, ensuring more stable learning even with imperfect reward models.
  Empirical results demonstrate that our framework consistently outperforms
  traditional RLHF across diverse benchmarks, showing improved accuracy and
  long-term stability. We also provide a theoretical analysis, demonstrating that
  reward-robust RLHF approaches the stability of constant reward settings, which
  proves to be effective in a stochastic-case analysis. Together, these
  contributions highlight the framework potential to enhance both the performance
  and stability of LLM alignment with RLHF.
paper_authors._id: 66f395d0685ee4f1ba10ff9d
paper_authors.name: Yuzi Yan
paper_authors._id: 66f395d0685ee4f1ba10ff9e
paper_authors.name: Xingzhou Lou
paper_authors._id: 66f395d0685ee4f1ba10ff9f
paper_authors.name: Jialian Li
paper_authors._id: 66f395d0685ee4f1ba10ffa0
paper_authors.name: Yiping Zhang
paper_authors._id: 66f395d0685ee4f1ba10ffa1
paper_authors.name: Jian Xie
paper_authors._id: 66f395d0685ee4f1ba10ffa2
paper_authors.name: Chao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:27:07.743Z
paper_authors._id: 64ba0f8d842aa47891cb972b
paper_authors.avatarUrl: /avatars/3bcacd9b778a146e88e20887b0b00720.svg
paper_authors.fullname: Chao Yu
paper_authors.type: user
paper_authors.user: zoeyuchao
paper_authors._id: 66f395d0685ee4f1ba10ffa3
paper_authors.name: Yu Wang
paper_authors._id: 66f395d0685ee4f1ba10ffa4
paper_authors.name: Dong Yan
paper_authors._id: 66f395d0685ee4f1ba10ffa5
paper_authors.name: Yuan Shen
paper_title: Reward-Robust RLHF in LLMs
paper_summary: As Large Language Models (LLMs) continue to progress toward more advanced
  forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
  increasingly seen as a key pathway toward achieving Artificial General
  Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
  alignment methods introduces significant challenges due to the inherent
  instability and imperfections of Reward Models (RMs), which can lead to
  critical issues such as reward hacking and misalignment with human intentions.
  In this paper, we introduce a reward-robust RLHF framework aimed at addressing
  these fundamental challenges, paving the way for more reliable and resilient
  learning in LLMs. Our approach introduces a novel optimization objective that
  carefully balances performance and robustness by incorporating Bayesian Reward
  Model Ensembles (BRME) to model the uncertainty set of reward functions. This
  allows the framework to integrate both nominal performance and minimum reward
  signals, ensuring more stable learning even with imperfect reward models.
  Empirical results demonstrate that our framework consistently outperforms
  traditional RLHF across diverse benchmarks, showing improved accuracy and
  long-term stability. We also provide a theoretical analysis, demonstrating that
  reward-robust RLHF approaches the stability of constant reward settings, which
  proves to be effective in a stochastic-case analysis. Together, these
  contributions highlight the framework potential to enhance both the performance
  and stability of LLM alignment with RLHF.
paper_ai_summary: A reward-robust RLHF framework using Bayesian Reward Model Ensembles enhances the stability and performance of LLM alignment with RLHF.
paper_ai_keywords: Large Language Models RLHF Artificial General Intelligence Reinforcement Learning from Human Feedback Reward Models reward hacking misalignment Bayesian Reward Model Ensembles BRME