title: Show, Don't Tell: Aligning Language Models with Demonstrated Feedback
summary: Language models are aligned to emulate the collective voice of many,
  resulting in outputs that align with no one in particular. Steering LLMs away
  from generic output is possible through supervised finetuning or RLHF, but
  requires prohibitively large datasets for new ad-hoc tasks. We argue that it is
  instead possible to align an LLM to a specific setting by leveraging a very
  small number (<10) of demonstrations as feedback. Our method, Demonstration
  ITerated Task Optimization (DITTO), directly aligns language model outputs to a
  user's demonstrated behaviors. Derived using ideas from online imitation
  learning, DITTO cheaply generates online comparison data by treating users'
  demonstrations as preferred over output from the LLM and its intermediate
  checkpoints. We evaluate DITTO's ability to learn fine-grained style and task
  alignment across domains such as news articles, emails, and blog posts.
  Additionally, we conduct a user study soliciting a range of demonstrations from
  participants (N=16). Across our benchmarks and user study, we find that
  win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and
  other self-play methods by an average of 19% points. By using demonstrations as
  feedback directly, DITTO offers a novel method for effective customization of
  LLMs.
paper_authors._id: 665e92c6ebc294bd6f621f2a
paper_authors.name: Omar Shaikh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:23:02.619Z
paper_authors._id: 632601d42a0af3bd4522aed7
paper_authors.avatarUrl: /avatars/4c02961e3a944c026c6444094d4642aa.svg
paper_authors.fullname: Omar Shaikh
paper_authors.type: user
paper_authors.user: oshaikh13
paper_authors._id: 665e92c6ebc294bd6f621f2b
paper_authors.name: Michelle Lam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:23:16.657Z
paper_authors._id: 6396bd9be794f987a771134b
paper_authors.avatarUrl: /avatars/d09544d56d1d77c529eb527b9bb79650.svg
paper_authors.fullname: Michelle Lam
paper_authors.type: user
paper_authors.user: michelle123lam
paper_authors._id: 665e92c6ebc294bd6f621f2c
paper_authors.name: Joey Hejna
paper_authors._id: 665e92c6ebc294bd6f621f2d
paper_authors.name: Yijia Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:24:45.822Z
paper_authors._id: 6171514808b4ddb7392f2eb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
paper_authors.fullname: Yijia Shao
paper_authors.type: user
paper_authors.user: EchoShao8899
paper_authors._id: 665e92c6ebc294bd6f621f2e
paper_authors.name: Michael Bernstein
paper_authors._id: 665e92c6ebc294bd6f621f2f
paper_authors.name: Diyi Yang
paper_title: Show, Don't Tell: Aligning Language Models with Demonstrated Feedback
paper_summary: Language models are aligned to emulate the collective voice of many,
  resulting in outputs that align with no one in particular. Steering LLMs away
  from generic output is possible through supervised finetuning or RLHF, but
  requires prohibitively large datasets for new ad-hoc tasks. We argue that it is
  instead possible to align an LLM to a specific setting by leveraging a very
  small number (<10) of demonstrations as feedback. Our method, Demonstration
  ITerated Task Optimization (DITTO), directly aligns language model outputs to a
  user's demonstrated behaviors. Derived using ideas from online imitation
  learning, DITTO cheaply generates online comparison data by treating users'
  demonstrations as preferred over output from the LLM and its intermediate
  checkpoints. We evaluate DITTO's ability to learn fine-grained style and task
  alignment across domains such as news articles, emails, and blog posts.
  Additionally, we conduct a user study soliciting a range of demonstrations from
  participants (N=16). Across our benchmarks and user study, we find that
  win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and
  other self-play methods by an average of 19% points. By using demonstrations as
  feedback directly, DITTO offers a novel method for effective customization of
  LLMs.
paper_ai_keywords: language models (LLMs) supervised finetuning RLHF demonstrations Demonstration ITerated Task Optimization (DITTO) online imitation learning comparison data intermediate checkpoints few-shot prompting self-play methods