zen and
used to train the LLM using RL, e.g., via PPO [Schulman et al., 2017]. A recent alternative
is to avoid training the reward model at all, and directly use human preferences to train the
LLM, as in Direct Preference Optimization [DPO; Rafailov et al., 2023]. In both cases, the
approach is bottlenecked by the size and quality of the human preference data, and in the
case of RLHF the quality of the frozen reward model trained from them as well.
In this work, we instead propose to train a self-improving