Show, Don't Tell: Aligning Language Models with Demonstrated Feedback
Language models are aligned to emulate the collective voice of many,
resulting in outputs that align with no one in particular. Steering LLMs away
from generic output is possible through supervised finetuning or RLHF, but
requires prohibitively large datasets for new ad-hoc tasks. We argue that it is
instead possible to align an LLM to a specific setting by leveraging a very
small number (<10) of demonstrations as feedback. Our method, Demonstration
ITerated Task Optimization (DITTO), directly aligns language model outputs to a
user's demonstrated behaviors. Derived using ideas from online imitation
learning, DITTO cheaply generates online comparison data by treating users'
demonstrations as preferred over output from the LLM and its intermediate
checkpoints. We evaluate DITTO's ability to learn fine-grained style and task
alignment across domains such as news articles, emails, and blog posts.
Additionally, we conduct a user study soliciting a range of demonstrations from
participants (N=16). Across our benchmarks and user study, we find that
win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and
other self-play methods by an average of 19% points. By using demonstrations as
feedback directly, DITTO offers a novel method for effective customization of
LLMs.
665e92c6ebc294bd6f621f2a
Omar Shaikh
admin_assigned
2024-06-04T08:23:02.619Z
632601d42a0af3bd4522aed7
/avatars/4c02961e3a944c026c6444094d4642aa.svg
Omar Shaikh
user
oshaikh13
665e92c6ebc294bd6f621f2b
Michelle Lam
admin_assigned
2024-06-04T08:23:16.657Z
6396bd9be794f987a771134b
/avatars/d09544d56d1d77c529eb527b9bb79650.svg
Michelle Lam
user
michelle123lam
665e92c6ebc294bd6f621f2c
Joey Hejna
665e92c6ebc294bd6f621f2d
Yijia Shao
claimed_verified
2024-09-05T07:24:45.822Z
6171514808b4ddb7392f2eb0
https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
Yijia Shao
user
EchoShao8899
665e92c6ebc294bd6f621f2e
Michael Bernstein
665e92c6ebc294bd6f621f2f
Diyi Yang
Show, Don't Tell: Aligning Language Models with Demonstrated Feedback
Language models are aligned to emulate the collective voice of many,
resulting in outputs that align with no one in particular. Steering LLMs away
from generic output is possible through supervised finetuning or RLHF, but
requires prohibitively large datasets for new ad-hoc tasks. We argue that it is
instead possible to align an LLM to a specific setting by leveraging a very
small number (<10) of demonstrations as feedback. Our method, Demonstration
ITerated Task Optimization (DITTO), directly aligns language model outputs to a
user's demonstrated behaviors. Derived using ideas from online imitation
learning, DITTO cheaply generates online comparison data by treating users'
demonstrations as preferred over output from the LLM and its intermediate
checkpoints. We evaluate DITTO's ability to learn fine-grained style and task
alignment across domains such as news articles, emails, and blog posts.
Additionally, we conduct a user study soliciting a range of demonstrations from
participants (N=16). Across our benchmarks and user study, we find that
win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and
other self-play methods by an average of 19% points. By using demonstrations as
feedback directly, DITTO offers a novel method for effective customization of
LLMs.
Demonstration ITerated Task Optimization (DITTO) aligns language models to specific settings using very few demonstrations, outperforming few-shot prompting and supervised fine-tuning across various domains.
LLMs supervised finetuning RLHF DITTO online imitation learning few-shot prompting task alignment news articles emails blog posts user study win-rates self-play methods