Next, we evaluate fine-tuning performance of DPO on summarization and single-turn dialogue. For
summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human
preferences [38], and prior work has found that fine-tuning LMs using PPO on human preferences
to provide more effective summaries. We evaluate different methods by sampling completions on
the test split of TL;DR summarization dataset, and computing the average win rate against reference
completions in the test set. The completions for all methods are sampled at temperatures varying
from 0.0 to 1.0, and the win rates are shown in Figure 2 (right). DPO, PPO and Preferred-FT all
fine-tune the same GPT-J SFT model4. We find that DPO has a win rate of approximately 61% at a
temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature
of 0.0. DPO also achieves a higher maximum win rate compared to the best of N baseline. We
Direct Preference Optimization