 PPO on
AlpacaFarm [Dubois et al., 2023]. PCO uses an iterative training approach similar to the
one in our work, except with a fixed reward model, and also showed that Iterative DPO
improves over DPO using the same scheme.
Reinforcement Learning from AI Feedback (RLAIF)
Constitutional AI [Bai et al.,
2022b] uses an LLM to give feedback and refine responses, and uses this data to train a
(separate, fixed) reward model. This is then used to perform “RL from AI Feedback” (RLAIF).
Lee et al. [2023] compare RLA