2. Related Work

Aligning Large Language Models LLMs are typically
aligned to human preferences using supervised fine-tuning
on demonstration data, followed by RLHF. RLHF con-
sists of training a reward function from comparison data on
model outputs to represent human preferences and then us-
ing reinforcement learning to align the policy model. Prior
work [4, 26, 29, 47] has used policy-gradient methods [27,
38] to this end. These methods are successful, but expen-
sive and require extensive hyperparameter tuning [34, 59],
and can be prone to reward hacking [10, 12, 41]. Alter-
native approaches sample base model answers and select
based on predicted rewards [3, 5, 14] to use for super-
vised training [2, 16, 50]. Methods that fine-tune the pol-
icy model directly on feedback data [1, 10], or utilize a
ranking loss on preference data to directly train the pol-
icy model [33, 49, 57, 58] have emerged. The latter set of
DiffusionModelAlignmentUsing Direct Preference Optimization