title: Direct Nash Optimization: Teaching Language Models to Self-Improve with
    General Preferences
summary: This paper studies post-training large language models (LLMs) using
  preference feedback from a powerful oracle to help a model iteratively improve
  over itself. The typical approach for post-training LLMs involves Reinforcement
  Learning from Human Feedback (RLHF), which traditionally separates reward
  learning and subsequent policy optimization. However, such a reward
  maximization approach is limited by the nature of "point-wise" rewards (such as
  Bradley-Terry model), which fails to express complex intransitive or cyclic
  preference relations. While advances on RLHF show reward learning and policy
  optimization can be merged into a single contrastive objective for stability,
  they yet still remain tethered to the reward maximization framework. Recently,
  a new wave of research sidesteps the reward maximization presumptions in favor
  of directly optimizing over "pair-wise" or general preferences. In this paper,
  we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm
  that marries the simplicity and stability of contrastive learning with
  theoretical generality from optimizing general preferences. Because DNO is a
  batched on-policy algorithm using a regression-based objective, its
  implementation is straightforward and efficient. Moreover, DNO enjoys monotonic
  improvement across iterations that help it improve even over a strong teacher
  (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model
  aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of
  33% on AlpacaEval 2.0 (even after controlling for response length), an absolute
  gain of 26% (7% to 33%) over the initializing model. It outperforms models with
  far more parameters, including Mistral Large, Self-Rewarding LM (70B
  parameters), and older versions of GPT-4.
paper_authors._id: 66135070174b378a723d9803
paper_authors.name: Corby Rosset
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-08T02:03:29.225Z
paper_authors._id: 64ec43147e2ec711a761c594
paper_authors.avatarUrl: /avatars/e4fc3de141ff026eee6eaa0b91d34ff1.svg
paper_authors.fullname: corby
paper_authors.type: user
paper_authors.user: corbyrosset
paper_authors._id: 66135070174b378a723d9804
paper_authors.name: Ching-An Cheng
paper_authors._id: 66135070174b378a723d9805
paper_authors.name: Arindam Mitra
paper_authors._id: 66135070174b378a723d9806
paper_authors.name: Michael Santacroce
paper_authors._id: 66135070174b378a723d9807
paper_authors.name: Ahmed Awadallah
paper_authors._id: 66135070174b378a723d9808
paper_authors.name: Tengyang Xie
paper_title: Direct Nash Optimization: Teaching Language Models to Self-Improve with
    General Preferences
paper_summary: This paper studies post-training large language models (LLMs) using
  preference feedback from a powerful oracle to help a model iteratively improve
  over itself. The typical approach for post-training LLMs involves Reinforcement
  Learning from Human Feedback (RLHF), which traditionally separates reward
  learning and subsequent policy optimization. However, such a reward
  maximization approach is limited by the nature of "point-wise" rewards (such as
  Bradley-Terry model), which fails to express complex intransitive or cyclic
  preference relations. While advances on RLHF show reward learning and policy
  optimization can be merged into a single contrastive objective for stability,
  they yet still remain tethered to the reward maximization framework. Recently,
  a new wave of research sidesteps the reward maximization presumptions in favor
  of directly optimizing over "pair-wise" or general preferences. In this paper,
  we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm
  that marries the simplicity and stability of contrastive learning with
  theoretical generality from optimizing general preferences. Because DNO is a
  batched on-policy algorithm using a regression-based objective, its
  implementation is straightforward and efficient. Moreover, DNO enjoys monotonic
  improvement across iterations that help it improve even over a strong teacher
  (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model
  aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of
  33% on AlpacaEval 2.0 (even after controlling for response length), an absolute
  gain of 26% (7% to 33%) over the initializing model. It outperforms models with
  far more parameters, including Mistral Large, Self-Rewarding LM (70B
  parameters), and older versions of GPT-4.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reward learning policy optimization point-wise rewards Bradley-Terry model contrastive objective pair-wise preferences general preferences Direct Nash Optimization (DNO) contrastive learning on-policy algorithm regression-based objective monotonic improvement Orca-2.5 AlpacaEval 2.0 GPT-4-Turbo GPT-4 Mistral Large Self-Rewarding LM