title: Scaling Laws for Reward Model Overoptimization in Direct Alignment
    Algorithms
summary: Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
  recent success of Large Language Models (LLMs), however, it is often a complex
  and brittle process. In the classical RLHF framework, a reward model is first
  trained to represent human preferences, which is in turn used by an online
  reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue
  with such methods is reward over-optimization or reward hacking,
  where performance as measured by the learned proxy reward model increases, but
  true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs)
  like Direct Preference Optimization have emerged as alternatives to the
  classical RLHF pipeline by circumventing the reward modeling phase. However,
  although DAAs do not use a separate proxy reward model, they still commonly
  deteriorate from over-optimization. While the so-called reward hacking
  phenomenon is not well-defined for DAAs, we still uncover similar trends: at
  higher KL budgets, DAA algorithms exhibit similar degradation patterns to their
  classic RLHF counterparts. In particular, we find that DAA methods deteriorate
  not only across a wide range of KL budgets but also often before even a single
  epoch of the dataset is completed. Through extensive empirical experimentation,
  this work formulates and formalizes the reward over-optimization or hacking
  problem for DAAs and explores its consequences across objectives, training
  regimes, and model scales.
paper_authors._id: 66612cac8585c09f28721245
paper_authors.name: Rafael Rafailov
paper_authors._id: 66612cac8585c09f28721246
paper_authors.name: Yaswanth Chittepu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-06T12:15:24.221Z
paper_authors._id: 6616d47e963a36b202a2ff7c
paper_authors.avatarUrl: /avatars/1bb03e24954d67b709827fe7af7a0634.svg
paper_authors.fullname: Yaswanth Chittepu
paper_authors.type: user
paper_authors.user: yaswanthchittepu
paper_authors._id: 66612cac8585c09f28721247
paper_authors.name: Ryan Park
paper_authors._id: 66612cac8585c09f28721248
paper_authors.name: Harshit Sikchi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-06T19:18:49.072Z
paper_authors._id: 6304559b1dd5d3c6248878f1
paper_authors.avatarUrl: /avatars/50ad7ffcc226c9ed7c6a78634a5fc8b4.svg
paper_authors.fullname: Harshit Sikchi
paper_authors.type: user
paper_authors.user: hsikchi
paper_authors._id: 66612cac8585c09f28721249
paper_authors.name: Joey Hejna
paper_authors._id: 66612cac8585c09f2872124a
paper_authors.name: Bradley Knox
paper_authors._id: 66612cac8585c09f2872124b
paper_authors.name: Chelsea Finn
paper_authors._id: 66612cac8585c09f2872124c
paper_authors.name: Scott Niekum
paper_title: Scaling Laws for Reward Model Overoptimization in Direct Alignment
    Algorithms
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
  recent success of Large Language Models (LLMs), however, it is often a complex
  and brittle process. In the classical RLHF framework, a reward model is first
  trained to represent human preferences, which is in turn used by an online
  reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue
  with such methods is reward over-optimization or reward hacking,
  where performance as measured by the learned proxy reward model increases, but
  true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs)
  like Direct Preference Optimization have emerged as alternatives to the
  classical RLHF pipeline by circumventing the reward modeling phase. However,
  although DAAs do not use a separate proxy reward model, they still commonly
  deteriorate from over-optimization. While the so-called reward hacking
  phenomenon is not well-defined for DAAs, we still uncover similar trends: at
  higher KL budgets, DAA algorithms exhibit similar degradation patterns to their
  classic RLHF counterparts. In particular, we find that DAA methods deteriorate
  not only across a wide range of KL budgets but also often before even a single
  epoch of the dataset is completed. Through extensive empirical experimentation,
  this work formulates and formalizes the reward over-optimization or hacking
  problem for DAAs and explores its consequences across objectives, training
  regimes, and model scales.
paper_ai_summary: The study investigates reward over-optimization and hacking in Direct Alignment Algorithms, finding similarities to classical Reinforcement Learning from Human Feedback methods, and formalizes this problem for DAAs.
paper_ai_keywords: Reinforcement Learning from Human Feedback Large Language Models reward over-optimization reward hacking Direct Preference Optimization Direct Alignment Algorithms KL budgets