7.2 Synthetic Supervision

Similar to our work in Section 4, Gao et al. (2022) use a large reward model to
supervise the training of smaller models. They study the over-optimization that
occurs during RLHF, with experiments that require large quantities of human
preference data. To work around this challenge, they use a gold-standard reward
model to replace human feedback. Our use of a large-scale reward model to
supervise smaller reward models shares similarities with their approach.

7.3 Natural Language Reasoning

Several recent studies that have examined the reasoning ability of large language
models are implicitly relevant to our work. Lewkowycz et al. (2022) showed that
finetuning models on a large corpus of technical content led to significantly im-
proved performance on MATH. Wang et al. (2022) showed that self-consistency

12
Let’s Verify Step by Step