title: Reward-Augmented Decoding: Efficient Controlled Text Generation With a
    Unidirectional Reward Model
summary: While large language models have proven effective in a huge range of
  downstream applications, they often generate text that is problematic or lacks
  a desired attribute. In this paper, we introduce Reward-Augmented Decoding
  (RAD), a text generation procedure that uses a small unidirectional reward
  model to encourage a language model to generate text that has certain
  properties. Specifically, RAD uses the reward model to score generations as
  they are produced and rescales sampling probabilities to favor high-reward
  tokens. By using a unidirectional reward model, RAD can cache activations from
  prior generation steps to decrease computational overhead. Through experiments
  on generating non-toxic and sentiment-controlled text, we demonstrate that RAD
  performs best among methods that change only the generation procedure and
  matches the performance of state-of-the-art methods that involve re-training
  the language model. We further validate that RAD is effective on very large
  language models while incurring a minimal computational overhead.
paper_authors._id: 652deaa839b7a1b6c84fe9a0
paper_authors.name: Haikang Deng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-17T05:35:45.468Z
paper_authors._id: 624cf6ba5ee9cb3d45588971
paper_authors.avatarUrl: /avatars/f0fc5e353951d3e501542dab559bbdb5.svg
paper_authors.fullname: Haikang Deng
paper_authors.type: user
paper_authors.user: hk
paper_authors._id: 652deaa839b7a1b6c84fe9a1
paper_authors.name: Colin Raffel
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-17T02:00:09.545Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_title: Reward-Augmented Decoding: Efficient Controlled Text Generation With a
    Unidirectional Reward Model
paper_summary: While large language models have proven effective in a huge range of
  downstream applications, they often generate text that is problematic or lacks
  a desired attribute. In this paper, we introduce Reward-Augmented Decoding
  (RAD), a text generation procedure that uses a small unidirectional reward
  model to encourage a language model to generate text that has certain
  properties. Specifically, RAD uses the reward model to score generations as
  they are produced and rescales sampling probabilities to favor high-reward
  tokens. By using a unidirectional reward model, RAD can cache activations from
  prior generation steps to decrease computational overhead. Through experiments
  on generating non-toxic and sentiment-controlled text, we demonstrate that RAD
  performs best among methods that change only the generation procedure and
  matches the performance of state-of-the-art methods that involve re-training
  the language model. We further validate that RAD is effective on very large
  language models while incurring a minimal computational overhead.
paper_ai_summary: Reward-Augmented Decoding (RAD) enhances text generation by using a unidirectional reward model to favor desirable text attributes, achieving performance comparable to re-training methods with minimal computational cost.
paper_ai_keywords: Reward-Augmented Decoding RAD reward model unidirectional caching computational overhead non-toxic text sentiment-controlled text large language models