∗Equal contribution; more junior authors listed earlier.

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Figure 1: DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods
for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and
human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward.
In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification
objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.
Direct Preference Optimization