title: Iterative Nash Policy Optimization: Aligning LLMs with General
    Preferences via No-Regret Learning
summary: Reinforcement Learning with Human Feedback (RLHF) has achieved great success
  in aligning large language models (LLMs) with human preferences. Prevalent RLHF
  approaches are reward-based, following the Bradley-Terry (BT) model assumption,
  which may not fully capture the complexity of human preferences. In this paper,
  we explore RLHF under a general preference framework and approach it from a
  game-theoretic perspective. Specifically, we formulate the problem as a
  two-player game and propose a novel algorithm, iterative Nash policy
  optimization (INPO). The key idea is to let the policy play against itself via
  no-regret learning, thereby approximating the Nash policy. Unlike previous
  methods, INPO bypasses the need for estimating the expected win rate for
  individual responses, which typically incurs high computational or annotation
  costs. Instead, we introduce a new loss objective that is directly minimized
  over a preference dataset. We provide theoretical analysis for our approach and
  demonstrate its effectiveness through experiments on various representative
  benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
  length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
  Arena-Hard, showing substantial improvement over the state-of-the-art iterative
  algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
  ablation study highlights the benefits of incorporating KL regularization for
  response length control.
paper_authors._id: 668352f81d6b7ca699ee42dd
paper_authors.name: Yuheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:39:09.289Z
paper_authors._id: 6645466c8899f25c56390ee0
paper_authors.avatarUrl: /avatars/4201db32587b67ffa9ddbf08385008f6.svg
paper_authors.fullname: Yuheng Zhang
paper_authors.type: user
paper_authors.user: MatouK98
paper_authors._id: 668352f81d6b7ca699ee42de
paper_authors.name: Dian Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:41:14.825Z
paper_authors._id: 62d58fd53bf5e059f7cc3245
paper_authors.avatarUrl: /avatars/7a4f3ee4a37245f67efd26749d66a706.svg
paper_authors.fullname: Dian Yu
paper_authors.type: user
paper_authors.user: yudian
paper_authors._id: 668352f81d6b7ca699ee42df
paper_authors.name: Baolin Peng
paper_authors._id: 668352f81d6b7ca699ee42e0
paper_authors.name: Linfeng Song
paper_authors._id: 668352f81d6b7ca699ee42e1
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:29:06.324Z
paper_authors._id: 6414061f996b2e426f23df48
paper_authors.avatarUrl: /avatars/db0af49c5f017f7c455a076b8b2212b6.svg
paper_authors.fullname: Ye Tian
paper_authors.type: user
paper_authors.user: yetian
paper_authors._id: 668352f81d6b7ca699ee42e2
paper_authors.name: Mingyue Huo
paper_authors._id: 668352f81d6b7ca699ee42e3
paper_authors.name: Nan Jiang
paper_authors._id: 668352f81d6b7ca699ee42e4
paper_authors.name: Haitao Mi
paper_authors._id: 668352f81d6b7ca699ee42e5
paper_authors.name: Dong Yu
paper_title: Iterative Nash Policy Optimization: Aligning LLMs with General
    Preferences via No-Regret Learning
paper_summary: Reinforcement Learning with Human Feedback (RLHF) has achieved great success
  in aligning large language models (LLMs) with human preferences. Prevalent RLHF
  approaches are reward-based, following the Bradley-Terry (BT) model assumption,
  which may not fully capture the complexity of human preferences. In this paper,
  we explore RLHF under a general preference framework and approach it from a
  game-theoretic perspective. Specifically, we formulate the problem as a
  two-player game and propose a novel algorithm, iterative Nash policy
  optimization (INPO). The key idea is to let the policy play against itself via
  no-regret learning, thereby approximating the Nash policy. Unlike previous
  methods, INPO bypasses the need for estimating the expected win rate for
  individual responses, which typically incurs high computational or annotation
  costs. Instead, we introduce a new loss objective that is directly minimized
  over a preference dataset. We provide theoretical analysis for our approach and
  demonstrate its effectiveness through experiments on various representative
  benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
  length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
  Arena-Hard, showing substantial improvement over the state-of-the-art iterative
  algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
  ablation study highlights the benefits of incorporating KL regularization for
  response length control.
paper_ai_keywords: Reinforcement Learning with Human Feedback (RLHF) Bradley-Terry (BT) model game-theoretic perspective two-player game iterative Nash policy optimization (INPO) no-regret learning Nash policy preference dataset length-controlled win rate AlpacaEval 2.0 Arena-Hard iterative algorithm KL regularization