D.2 Sample Responses and GPT-4 Judgments

In this section, we present examples of comparisons between DPO and the baseline (PPO temp 0. for
summarization, and the ground truth chosen response for dialogue). See Tables 4-6 for summarization
examples, and Tables 7-10 for dialogue examples.

22

Figure 4: Best of N baseline for N = {1, 4, 16, 64, 128}. Performance plateaus after roughly 64-128 samples.

Prompt

DPO

PPO

Judgment
Direct Preference Optimization