we will show that the RL-based objective used by existing methods can be optimized exactly with a
simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.
At a high level, existing methods instill the desired behaviors into a language model using curated
sets of human preferences representing the types of behaviors that humans find safe and helpful. This
preference learning stage occurs after an initial stage of large-scale unsupervised pre-training on
a large text dataset. While the most straightforward approach to preference learning is supervised
fine-tuning on human demonstrations of high quality responses, the most successful class of methods
is reinforcement learning from human (or AI) feedback (RLHF/RLAIF; [12, 2]). RLHF methods fit
a reward model to a dataset of human preferences and then use RL to optimize a language model
policy to produce responses assigned high reward without drifting excessively far from the original
Direct Preference Optimization