All authors provided valuable contributions to designing, analyzing, and iterating on experiments,
writing and editing the paper, and generally managing the project’s progress.
RR proposed using autoregressive reward models in discussions with EM; derived the DPO objective;
proved the theoretical properties of the algorithm and wrote the relevant sections and appendices. He
also suggested and helped with organizing experiments and contributed some of the PPO and reward
learning baselines.
AS initiated the discussion on using weighted regression methods as an alternative to PPO; initiated
project-related organization, wrote initial analysis connecting DPO with weighted regression and
unlikelihood; design and iterations of DPO + baseline implementations, initial exploratory exper-
iments for DPO; substantial experiment organization and design (datasets, baselines, evaluation);
led model training and evaluation for controlled sentiment generation and summarization; design
Direct Preference Optimization