and KL-divergence from the reference policy; this frontier is computable because we have acccess to
the ground-truth reward function (a sentiment classifier). However, in the real world, the ground truth
reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline
policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness
in the summarization and single-turn dialogue settings, respectively. For summarization, we use
reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the
Direct Preference Optimization