title: RLVF: Learning from Verbal Feedback without Overgeneralization
summary: The diversity of contexts in which large language models (LLMs) are deployed
  requires the ability to modify or customize default model behaviors to
  incorporate nuanced requirements and preferences. A convenient interface to
  specify such model adjustments is high-level verbal feedback, such as "Don't
  use emojis when drafting emails to my boss." However, while writing high-level
  feedback is far simpler than collecting annotations for reinforcement learning
  from human feedback (RLHF), we find that simply prompting a model with such
  feedback leads to overgeneralization of the feedback to contexts where it is
  not relevant. We study the problem of incorporating verbal feedback without
  such overgeneralization, inspiring a new method Contextualized Critiques with
  Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level
  feedback to generate a small synthetic preference dataset specifying how the
  feedback should (and should not) be applied. It then fine-tunes the model in
  accordance with the synthetic preference data while minimizing the divergence
  from the original model for prompts where the feedback does not apply. Our
  experimental results indicate that our approach effectively applies verbal
  feedback to relevant scenarios while preserving existing behaviors for other
  contexts. For both human- and GPT-4-generated high-level feedback, C3PO
  effectively adheres to the given feedback comparably to in-context baselines
  while reducing overgeneralization by 30%.
paper_authors._id: 65d2bc1b351255ba48d156a7
paper_authors.name: Moritz Stephan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-19T02:27:17.754Z
paper_authors._id: 65263403ab48a81abd5847d5
paper_authors.avatarUrl: /avatars/944765111baffb8a24a5b32fc3daab15.svg
paper_authors.fullname: Moritz Stephan
paper_authors.type: user
paper_authors.user: Moritz-Stephan
paper_authors._id: 65d2bc1b351255ba48d156a8
paper_authors.name: Alexander Khazatsky
paper_authors._id: 65d2bc1b351255ba48d156a9
paper_authors.name: Eric Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:24:54.095Z
paper_authors._id: 6514e9cbfc2301d62d7bf2ce
paper_authors.avatarUrl: /avatars/eba6bd6e2840d7b5dd735e6662020d0c.svg
paper_authors.fullname: Eric mitchell
paper_authors.type: user
paper_authors.user: Pain777
paper_authors._id: 65d2bc1b351255ba48d156aa
paper_authors.name: Annie S Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:25:34.622Z
paper_authors._id: 64c06c248137192a1e1e1428
paper_authors.avatarUrl: /avatars/a1fc70a8f28a45807beec09e413ee362.svg
paper_authors.fullname: Annie Chen
paper_authors.type: user
paper_authors.user: anniesch
paper_authors._id: 65d2bc1b351255ba48d156ab
paper_authors.name: Sheryl Hsu
paper_authors._id: 65d2bc1b351255ba48d156ac
paper_authors.name: Archit Sharma
paper_authors._id: 65d2bc1b351255ba48d156ad
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:26:14.353Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_title: RLVF: Learning from Verbal Feedback without Overgeneralization
paper_summary: The diversity of contexts in which large language models (LLMs) are deployed
  requires the ability to modify or customize default model behaviors to
  incorporate nuanced requirements and preferences. A convenient interface to
  specify such model adjustments is high-level verbal feedback, such as "Don't
  use emojis when drafting emails to my boss." However, while writing high-level
  feedback is far simpler than collecting annotations for reinforcement learning
  from human feedback (RLHF), we find that simply prompting a model with such
  feedback leads to overgeneralization of the feedback to contexts where it is
  not relevant. We study the problem of incorporating verbal feedback without
  such overgeneralization, inspiring a new method Contextualized Critiques with
  Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level
  feedback to generate a small synthetic preference dataset specifying how the
  feedback should (and should not) be applied. It then fine-tunes the model in
  accordance with the synthetic preference data while minimizing the divergence
  from the original model for prompts where the feedback does not apply. Our
  experimental results indicate that our approach effectively applies verbal
  feedback to relevant scenarios while preserving existing behaviors for other
  contexts. For both human- and GPT-4-generated high-level feedback, C3PO
  effectively adheres to the given feedback comparably to in-context baselines
  while reducing overgeneralization by 30%.
paper_ai_summary: A new method called C3PO is introduced to incorporate verbal feedback into large language models without overgeneralization by generating a synthetic preference dataset and fine-tuning the model accordingly.
paper_ai_keywords: large language models reinforcement learning from human feedback RLHF high-level verbal feedback overgeneralization Contextualized Critiques with Constrained Preference Optimization C3PO synthetic preference dataset fine-tuning divergence GPT-4