title: Controlled Decoding from Language Models
summary: We propose controlled decoding (CD), a novel off-policy reinforcement
  learning method to control the autoregressive generation from language models
  towards high reward outcomes. CD solves an off-policy reinforcement learning
  problem through a value function for the reward, which we call a prefix scorer.
  The prefix scorer is used at inference time to steer the generation towards
  higher reward outcomes. We show that the prefix scorer may be trained on
  (possibly) off-policy data to predict the expected reward when decoding is
  continued from a partially decoded response. We empirically demonstrate that CD
  is effective as a control mechanism on Reddit conversations corpus. We also
  show that the modularity of the design of CD makes it possible to control for
  multiple rewards, effectively solving a multi-objective reinforcement learning
  problem with no additional complexity. Finally, we show that CD can be applied
  in a novel blockwise fashion at inference-time, again without the need for any
  training-time changes, essentially bridging the gap between the popular
  best-of-K strategy and token-level reinforcement learning. This makes CD a
  promising approach for alignment of language models.
paper_authors._id: 653b17660157cf1d9c1261bb
paper_authors.name: Sidharth Mudgal
paper_authors._id: 653b17660157cf1d9c1261bc
paper_authors.name: Jong Lee
paper_authors._id: 653b17660157cf1d9c1261bd
paper_authors.name: Harish Ganapathy
paper_authors._id: 653b17660157cf1d9c1261be
paper_authors.name: YaGuang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-27T18:43:14.777Z
paper_authors._id: 642f957c4e7b53c5addcae64
paper_authors.avatarUrl: /avatars/0bb13eafdc0a80e1d030d570efaa7033.svg
paper_authors.fullname: YaGuang Li
paper_authors.type: user
paper_authors.user: kakaro0t
paper_authors._id: 653b17660157cf1d9c1261bf
paper_authors.name: Tao Wang
paper_authors._id: 653b17660157cf1d9c1261c0
paper_authors.name: Yanping Huang
paper_authors._id: 653b17660157cf1d9c1261c1
paper_authors.name: Zhifeng Chen
paper_authors._id: 653b17660157cf1d9c1261c2
paper_authors.name: Heng-Tze Cheng
paper_authors._id: 653b17660157cf1d9c1261c3
paper_authors.name: Michael Collins
paper_authors._id: 653b17660157cf1d9c1261c4
paper_authors.name: Trevor Strohman
paper_authors._id: 653b17660157cf1d9c1261c5
paper_authors.name: Jilin Chen
paper_authors._id: 653b17660157cf1d9c1261c6
paper_authors.name: Alex Beutel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:27:56.647Z
paper_authors._id: 636dc726ffbe479c979c3046
paper_authors.avatarUrl: /avatars/287f422bb0c5a26d8d6c0649ac83ba93.svg
paper_authors.fullname: Alex Beutel
paper_authors.type: user
paper_authors.user: alexbeutel
paper_authors._id: 653b17660157cf1d9c1261c7
paper_authors.name: Ahmad Beirami
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-06T10:57:30.766Z
paper_authors._id: 65482d45407bb19ff5adb8a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_etKjaaubh3osxp3R2MP7.jpeg
paper_authors.fullname: Ahmad Beirami
paper_authors.type: user
paper_authors.user: beirami
paper_title: Controlled Decoding from Language Models
paper_summary: We propose controlled decoding (CD), a novel off-policy reinforcement
  learning method to control the autoregressive generation from language models
  towards high reward outcomes. CD solves an off-policy reinforcement learning
  problem through a value function for the reward, which we call a prefix scorer.
  The prefix scorer is used at inference time to steer the generation towards
  higher reward outcomes. We show that the prefix scorer may be trained on
  (possibly) off-policy data to predict the expected reward when decoding is
  continued from a partially decoded response. We empirically demonstrate that CD
  is effective as a control mechanism on Reddit conversations corpus. We also
  show that the modularity of the design of CD makes it possible to control for
  multiple rewards, effectively solving a multi-objective reinforcement learning
  problem with no additional complexity. Finally, we show that CD can be applied
  in a novel blockwise fashion at inference-time, again without the need for any
  training-time changes, essentially bridging the gap between the popular
  best-of-K strategy and token-level reinforcement learning. This makes CD a
  promising approach for alignment of language models.
paper_ai_keywords: controlled decoding (CD) off-policy reinforcement learning value function prefix scorer autoregressive generation high reward outcomes multi-objective reinforcement learning blockwise fashion best-of-$K$ strategy token-level reinforcement learning alignment of language models