 computationally cheap). Finally, Chen et al. [2024] recently showed they can
avoid reward models entirely in an Iterative DPO-like framework by using human labels as
the winning response in a pair, and the last iteration’s generations as the losing response in
the pair. The authors note this has the limitation that once the model generations reach
human performance, they are bottlenecked. Further, each input prompt is required to have
a human annotated response, in contrast to our work.
Improving LLMs via 