title: Toward Evaluative Thinking: Meta Policy Optimization with Evolving
    Reward Models
summary: Reward-based alignment methods for large language models (LLMs) face two key
  limitations: vulnerability to reward hacking, where models exploit flaws in the
  reward signal; and reliance on brittle, labor-intensive prompt engineering when
  LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a
  framework that addresses these challenges by integrating a meta-reward model
  that dynamically refines the reward model's prompt throughout training. In MPO,
  the meta-reward model monitors the evolving training context and continuously
  adjusts the reward model's prompt to maintain high alignment, providing an
  adaptive reward signal that resists exploitation by the policy. This
  meta-learning approach promotes a more stable policy optimization, and greatly
  reduces the need for manual reward prompt design. It yields performance on par
  with or better than models guided by extensively hand-crafted reward prompts.
  Furthermore, we show that MPO maintains its effectiveness across diverse tasks,
  such as question answering and mathematical reasoning, without requiring
  specialized reward designs. Beyond standard RLAIF, MPO's meta-learning
  formulation is readily extensible to higher-level alignment frameworks.
  Overall, this method addresses theoretical and practical challenges in
  reward-based RL alignment for LLMs, paving the way for more robust and
  adaptable alignment strategies. The code and models will be publicly shared.
paper_authors._id: 68119750ff0764f3840a7f93
paper_authors.name: Zae Myung Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:55:58.262Z
paper_authors._id: 61e0c5053a1781f66b4e9aed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1642120523097-61e0c5053a1781f66b4e9aed.jpeg
paper_authors.fullname: Zae Myung Kim
paper_authors.type: user
paper_authors.user: zaemyung
paper_authors._id: 68119750ff0764f3840a7f94
paper_authors.name: Chanwoo Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:41:49.528Z
paper_authors._id: 63f4e18407640ba6fcf92870
paper_authors.avatarUrl: /avatars/9392cc42754ba06b32297f6678f58216.svg
paper_authors.fullname: Chanwoo Park
paper_authors.type: user
paper_authors.user: Chanwoo97
paper_authors._id: 68119750ff0764f3840a7f95
paper_authors.name: Vipul Raheja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:03:14.631Z
paper_authors._id: 60985a0547dc3dbf8a976607
paper_authors.avatarUrl: /avatars/3c37bf4b7c9db83a46af7c473ee4eb86.svg
paper_authors.fullname: Vipul Raheja
paper_authors.type: user
paper_authors.user: machineteacher
paper_authors._id: 68119750ff0764f3840a7f96
paper_authors.name: Dongyeop Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:03:26.612Z
paper_authors._id: 64356b40a4bd75c62cbc5926
paper_authors.avatarUrl: /avatars/5f4c603464e9c8ad613a3a25fa4cacbf.svg
paper_authors.fullname: Dongyeop Kang
paper_authors.type: user
paper_authors.user: dykang
paper_title: Toward Evaluative Thinking: Meta Policy Optimization with Evolving
    Reward Models
paper_summary: Reward-based alignment methods for large language models (LLMs) face two key
  limitations: vulnerability to reward hacking, where models exploit flaws in the
  reward signal; and reliance on brittle, labor-intensive prompt engineering when
  LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a
  framework that addresses these challenges by integrating a meta-reward model
  that dynamically refines the reward model's prompt throughout training. In MPO,
  the meta-reward model monitors the evolving training context and continuously
  adjusts the reward model's prompt to maintain high alignment, providing an
  adaptive reward signal that resists exploitation by the policy. This
  meta-learning approach promotes a more stable policy optimization, and greatly
  reduces the need for manual reward prompt design. It yields performance on par
  with or better than models guided by extensively hand-crafted reward prompts.
  Furthermore, we show that MPO maintains its effectiveness across diverse tasks,
  such as question answering and mathematical reasoning, without requiring
  specialized reward designs. Beyond standard RLAIF, MPO's meta-learning
  formulation is readily extensible to higher-level alignment frameworks.
  Overall, this method addresses theoretical and practical challenges in
  reward-based RL alignment for LLMs, paving the way for more robust and
  adaptable alignment strategies. The code and models will be publicly shared.
paper_ai_summary: MPO is a framework that dynamically refines reward signals in LLMs through meta-reward modeling, enhancing alignment and reducing reward hacking and prompt engineering needs.
paper_ai_keywords: reward-based alignment large language models reward hacking prompt engineering Meta Policy Optimization meta-reward model policy optimization reward-based RL alignment