emini Pro,
and GPT-4 0613. While only a preliminary study, this work opens the door
to the possibility of models that can continually improve in both axes.
1
Introduction
Aligning Large Language Models (LLMs) using human preference data can vastly improve
the instruction following performance of pretrained models [Ouyang et al., 2022, Bai et al.,
2022a]. The standard approach of Reinforcement Learning from Human Feedback (RLHF)
learns a reward model from these human preferences. The reward model is then fro