title: Nash Learning from Human Feedback
summary: Reinforcement learning from human feedback (RLHF) has emerged as the main
  paradigm for aligning large language models (LLMs) with human preferences.
  Typically, RLHF involves the initial step of learning a reward model from human
  feedback, often expressed as preferences between pairs of text generations
  produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by
  optimizing it to maximize the reward model through a reinforcement learning
  algorithm. However, an inherent limitation of current reward models is their
  inability to fully represent the richness of human preferences and their
  dependency on the sampling distribution.
    In this study, we introduce an alternative pipeline for the fine-tuning of
  LLMs using pairwise human feedback. Our approach entails the initial learning
  of a preference model, which is conditioned on two inputs given a prompt,
  followed by the pursuit of a policy that consistently generates responses
  preferred over those generated by any competing policy, thus defining the Nash
  equilibrium of this preference model. We term this approach Nash learning from
  human feedback (NLHF).
    In the context of a tabular policy representation, we present a novel
  algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  This algorithm produces a sequence of policies, with the last iteration
  converging to the regularized Nash equilibrium. Additionally, we explore
  parametric representations of policies and introduce gradient descent
  algorithms for deep-learning architectures. To demonstrate the effectiveness of
  our approach, we present experimental results involving the fine-tuning of a
  LLM for a text summarization task. We believe NLHF offers a compelling avenue
  for preference learning and policy optimization with the potential of advancing
  the field of aligning LLMs with human preferences.
paper_authors._id: 656e9a8c7ef5b6b3b85b7d35
paper_authors.name: Rémi Munos
paper_authors._id: 656e9a8c7ef5b6b3b85b7d36
paper_authors.name: Michal Valko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T19:42:37.801Z
paper_authors._id: 651e97156d92456bdf5ace6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651e97156d92456bdf5ace6b/KKfdZGPAcWPdqycp9SulH.jpeg
paper_authors.fullname: Michal Valko
paper_authors.type: user
paper_authors.user: misovalko
paper_authors._id: 656e9a8c7ef5b6b3b85b7d37
paper_authors.name: Daniele Calandriello
paper_authors._id: 656e9a8c7ef5b6b3b85b7d38
paper_authors.name: Mohammad Gheshlaghi Azar
paper_authors._id: 656e9a8c7ef5b6b3b85b7d39
paper_authors.name: Mark Rowland
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3a
paper_authors.name: Daniel Guo
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3b
paper_authors.name: Yunhao Tang
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3c
paper_authors.name: Matthieu Geist
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:53.085Z
paper_authors._id: 67bef153eaca0bef78ed5478
paper_authors.avatarUrl: /avatars/ab838c541bd170986370a99670981c2f.svg
paper_authors.fullname: Matthieu Geist
paper_authors.type: user
paper_authors.user: matthieu-geist
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3d
paper_authors.name: Thomas Mésnard
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3e
paper_authors.name: Andrea Michi
paper_authors._id: 656e9a8c7ef5b6b3b85b7d3f
paper_authors.name: Marco Selvi
paper_authors._id: 656e9a8c7ef5b6b3b85b7d40
paper_authors.name: Sertan Girgin
paper_authors._id: 656e9a8c7ef5b6b3b85b7d41
paper_authors.name: Nikola Momchev
paper_authors._id: 656e9a8c7ef5b6b3b85b7d42
paper_authors.name: Olivier Bachem
paper_authors._id: 656e9a8c7ef5b6b3b85b7d43
paper_authors.name: Daniel J. Mankowitz
paper_authors._id: 656e9a8c7ef5b6b3b85b7d44
paper_authors.name: Doina Precup
paper_authors._id: 656e9a8c7ef5b6b3b85b7d45
paper_authors.name: Bilal Piot
paper_title: Nash Learning from Human Feedback
paper_summary: Reinforcement learning from human feedback (RLHF) has emerged as the main
  paradigm for aligning large language models (LLMs) with human preferences.
  Typically, RLHF involves the initial step of learning a reward model from human
  feedback, often expressed as preferences between pairs of text generations
  produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by
  optimizing it to maximize the reward model through a reinforcement learning
  algorithm. However, an inherent limitation of current reward models is their
  inability to fully represent the richness of human preferences and their
  dependency on the sampling distribution.
    In this study, we introduce an alternative pipeline for the fine-tuning of
  LLMs using pairwise human feedback. Our approach entails the initial learning
  of a preference model, which is conditioned on two inputs given a prompt,
  followed by the pursuit of a policy that consistently generates responses
  preferred over those generated by any competing policy, thus defining the Nash
  equilibrium of this preference model. We term this approach Nash learning from
  human feedback (NLHF).
    In the context of a tabular policy representation, we present a novel
  algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  This algorithm produces a sequence of policies, with the last iteration
  converging to the regularized Nash equilibrium. Additionally, we explore
  parametric representations of policies and introduce gradient descent
  algorithms for deep-learning architectures. To demonstrate the effectiveness of
  our approach, we present experimental results involving the fine-tuning of a
  LLM for a text summarization task. We believe NLHF offers a compelling avenue
  for preference learning and policy optimization with the potential of advancing
  the field of aligning LLMs with human preferences.
paper_ai_summary: NLHF uses a preference model and mirror descent to fine-tune LLMs for text summarization, advancing alignment with human preferences.
paper_ai_keywords: reinforcement learning from human feedback (RLHF) large language models (LLMs) reward model pairwise human feedback preference model Nash equilibrium Nash-MD mirror descent parametric representations gradient descent deep-learning architectures text summarization