hese works, but distilled from human data. Nevertheless, this is commonly referred
to as RL from Human Feedback (RLHF). Methods such as Direct Preference Optimization
(DPO) [Rafailov et al., 2023] avoid training the reward model entirely, and instead directly
train the LLM using human preferences. Several other such competing methods exist as
well [Zhao et al., 2023, Zheng et al., 2023a, Yuan et al., 2023], including Pairwise Cringe
Optimization (PCO) [Xu et al., 2023], which was shown to outperform DPO and