title: Offline Regularised Reinforcement Learning for Large Language Models
    Alignment
summary: The dominant framework for alignment of large language models (LLM), whether
  through reinforcement learning from human feedback or direct preference
  optimisation, is to learn from preference data. This involves building datasets
  where each element is a quadruplet composed of a prompt, two independent
  responses (completions of the prompt) and a human preference between the two
  independent responses, yielding a preferred and a dis-preferred response. Such
  data is typically scarce and expensive to collect. On the other hand,
  single-trajectory datasets where each element is a triplet composed of a
  prompt, a response and a human feedback is naturally more abundant. The
  canonical element of such datasets is for instance an LLM's response to a
  user's prompt followed by a user's feedback such as a thumbs-up/down.
  Consequently, in this work, we propose DRO, or Direct Reward
  Optimisation, as a framework and associated algorithms that do not require
  pairwise preferences. DRO uses a simple mean-squared objective that can be
  implemented in various ways. We validate our findings empirically, using T5
  encoder-decoder language models, and show DRO's performance over selected
  baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that
  DRO is a simple and empirically compelling method for single-trajectory policy
  optimisation.
paper_authors._id: 665888d824f3d17e4bdc9c89
paper_authors.name: Pierre Harvey Richemond
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:07:07.227Z
paper_authors._id: 636eeea187545ca5a13bedbc
paper_authors.avatarUrl: /avatars/cdbb5e0a700b60a43c99f93f8e9ef2b4.svg
paper_authors.fullname: Pierre Richemond
paper_authors.type: user
paper_authors.user: PierreRichemond
paper_authors._id: 665888d824f3d17e4bdc9c8a
paper_authors.name: Yunhao Tang
paper_authors._id: 665888d824f3d17e4bdc9c8b
paper_authors.name: Daniel Guo
paper_authors._id: 665888d824f3d17e4bdc9c8c
paper_authors.name: Daniele Calandriello
paper_authors._id: 665888d824f3d17e4bdc9c8d
paper_authors.name: Mohammad Gheshlaghi Azar
paper_authors._id: 665888d824f3d17e4bdc9c8e
paper_authors.name: Rafael Rafailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:07:57.576Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 665888d824f3d17e4bdc9c8f
paper_authors.name: Bernardo Avila Pires
paper_authors._id: 665888d824f3d17e4bdc9c90
paper_authors.name: Eugene Tarassov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T14:24:11.912Z
paper_authors._id: 63280235bfc72a1d59d029c3
paper_authors.avatarUrl: /avatars/85be03ddbe48a0d7a12f8fd95eeb00a1.svg
paper_authors.fullname: Eugene Tarassov
paper_authors.type: user
paper_authors.user: etarassov
paper_authors._id: 665888d824f3d17e4bdc9c91
paper_authors.name: Lucas Spangher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:08:28.512Z
paper_authors._id: 64d6785736b05ef8ffcf8d63
paper_authors.avatarUrl: /avatars/7bb694f2f5bf55462fe8d954b48a6a0e.svg
paper_authors.fullname: Lucas Spangher
paper_authors.type: user
paper_authors.user: Lucas-mit-psfc
paper_authors._id: 665888d824f3d17e4bdc9c92
paper_authors.name: Will Ellsworth
paper_authors._id: 665888d824f3d17e4bdc9c93
paper_authors.name: Aliaksei Severyn
paper_authors._id: 665888d824f3d17e4bdc9c94
paper_authors.name: Jonathan Mallinson
paper_authors._id: 665888d824f3d17e4bdc9c95
paper_authors.name: Lior Shani
paper_authors._id: 665888d824f3d17e4bdc9c96
paper_authors.name: Gil Shamir
paper_authors._id: 665888d824f3d17e4bdc9c97
paper_authors.name: Rishabh Joshi
paper_authors._id: 665888d824f3d17e4bdc9c98
paper_authors.name: Tianqi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:09:13.411Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 665888d824f3d17e4bdc9c99
paper_authors.name: Remi Munos
paper_authors._id: 665888d824f3d17e4bdc9c9a
paper_authors.name: Bilal Piot
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-03T18:28:18.402Z
paper_authors._id: 67a10a285685d37e288d1ed0
paper_authors.avatarUrl: /avatars/456dcb7bfe61b2bb423b89c9d622c54c.svg
paper_authors.fullname: Bilal Piot
paper_authors.type: user
paper_authors.user: bilalpiot
paper_title: Offline Regularised Reinforcement Learning for Large Language Models
    Alignment
paper_summary: The dominant framework for alignment of large language models (LLM), whether
  through reinforcement learning from human feedback or direct preference
  optimisation, is to learn from preference data. This involves building datasets
  where each element is a quadruplet composed of a prompt, two independent
  responses (completions of the prompt) and a human preference between the two
  independent responses, yielding a preferred and a dis-preferred response. Such
  data is typically scarce and expensive to collect. On the other hand,
  single-trajectory datasets where each element is a triplet composed of a
  prompt, a response and a human feedback is naturally more abundant. The
  canonical element of such datasets is for instance an LLM's response to a
  user's prompt followed by a user's feedback such as a thumbs-up/down.
  Consequently, in this work, we propose DRO, or Direct Reward
  Optimisation, as a framework and associated algorithms that do not require
  pairwise preferences. DRO uses a simple mean-squared objective that can be
  implemented in various ways. We validate our findings empirically, using T5
  encoder-decoder language models, and show DRO's performance over selected
  baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that
  DRO is a simple and empirically compelling method for single-trajectory policy
  optimisation.
paper_ai_summary: Direct Reward Optimisation (DRO) framework uses single-trajectory datasets for large language model alignment, offering a simpler and effective alternative to preference-based methods.
paper_ai_keywords: reinforcement learning from human feedback direct preference optimisation preference data quadruplet triplet human preference Abundant Direct Reward Optimisation mean-squared objective T5 encoder-decoder Kahneman-Tversky Optimization single-trajectory policy optimisation