title: RL4F: Generating Natural Language Feedback with Reinforcement Learning
    for Repairing Model Outputs
summary: Despite their unprecedented success, even the largest language models make
  mistakes. Similar to how humans learn and improve using feedback, previous work
  proposed providing language models with natural language feedback to guide them
  in repairing their outputs. Because human-generated critiques are expensive to
  obtain, researchers have devised learned critique generators in lieu of human
  critics while assuming one can train downstream models to utilize generated
  feedback. However, this approach does not apply to black-box or limited access
  models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of
  large general-purpose language agents, fine-tuning is neither computationally
  nor spatially efficient as it results in multiple copies of the network. In
  this work, we introduce RL4F (Reinforcement Learning for Feedback), a
  multi-agent collaborative framework where the critique generator is trained to
  maximize end-task performance of GPT-3, a fixed model more than 200 times its
  size. RL4F produces critiques that help GPT-3 revise its outputs. We study
  three datasets for action planning, summarization and alphabetization and show
  improvements (~5% on average) in multiple text similarity metrics over strong
  baselines across all three tasks.
paper_authors._id: 6462e0b18bd7025ebe85297a
paper_authors.name: Afra Feyza Akyürek
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-23T18:14:37.075Z
paper_authors._id: 617b009a7b4dce0224d6b8fb
paper_authors.avatarUrl: /avatars/d42ebf8c75ec6236d9fcbb3d1e0ba0b7.svg
paper_authors.fullname: Afra Feyza Akyurek
paper_authors.type: user
paper_authors.user: feyzaakyurek
paper_authors._id: 6462e0b18bd7025ebe85297b
paper_authors.name: Ekin Akyürek
paper_authors._id: 6462e0b18bd7025ebe85297c
paper_authors.name: Aman Madaan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:37:35.555Z
paper_authors._id: 61bbc2b2d96f5d4111901dcb
paper_authors.avatarUrl: /avatars/83c826d5cad7bab27d33585cb7e64575.svg
paper_authors.fullname: Aman Madaan
paper_authors.type: user
paper_authors.user: amanmadaan
paper_authors._id: 6462e0b18bd7025ebe85297d
paper_authors.name: Ashwin Kalyan
paper_authors._id: 6462e0b18bd7025ebe85297e
paper_authors.name: Peter Clark
paper_authors._id: 6462e0b18bd7025ebe85297f
paper_authors.name: Derry Wijaya
paper_authors._id: 6462e0b18bd7025ebe852980
paper_authors.name: Niket Tandon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:38:29.616Z
paper_authors._id: 62e69a709bb6d949274ad32b
paper_authors.avatarUrl: /avatars/b6a796d12a1c1c708f5b46c9082cdee6.svg
paper_authors.fullname: Niket Tandon
paper_authors.type: user
paper_authors.user: nikett
paper_title: RL4F: Generating Natural Language Feedback with Reinforcement Learning
    for Repairing Model Outputs
paper_summary: Despite their unprecedented success, even the largest language models make
  mistakes. Similar to how humans learn and improve using feedback, previous work
  proposed providing language models with natural language feedback to guide them
  in repairing their outputs. Because human-generated critiques are expensive to
  obtain, researchers have devised learned critique generators in lieu of human
  critics while assuming one can train downstream models to utilize generated
  feedback. However, this approach does not apply to black-box or limited access
  models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of
  large general-purpose language agents, fine-tuning is neither computationally
  nor spatially efficient as it results in multiple copies of the network. In
  this work, we introduce RL4F (Reinforcement Learning for Feedback), a
  multi-agent collaborative framework where the critique generator is trained to
  maximize end-task performance of GPT-3, a fixed model more than 200 times its
  size. RL4F produces critiques that help GPT-3 revise its outputs. We study
  three datasets for action planning, summarization and alphabetization and show
  improvements (~5% on average) in multiple text similarity metrics over strong
  baselines across all three tasks.
paper_ai_keywords: Reinforcement Learning multi-agent collaborative framework critique generator end-task performance GPT-3 action planning summarization alphabetization text similarity descriptions