Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for
training language models from preferences. Our experiments show that DPO is at least as effective
as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as
sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.
Direct Preference Optimization