We instead focus exclusively on how to train the most reliable reward model
possible. We evaluate a reward model by its ability to perform best-of-N search
over uniformly sampled solutions from the generator. For each test problem we
select the solution ranked highest by the reward model, automatically grade it
based on its final answer, and report the fraction that are correct. A reward
model that is more reliable will select the correct solution more often.

2.2 Base Models

All large-scale models are finetuned from the base GPT-4 model (OpenAI, 2023).
This model has been pretrained solely to predict the next token; it has not been
pretrained with any Reinforcement Learning from Human Feedback (RLHF)
(Christiano et al., 2017). The small-scale base models are similar in design to
GPT-4, but they were pretrained with roughly 200 times less compute. As an
additional pretraining step, we finetune all models on a dataset of roughly 1.5B

3
Let’s Verify Step by Step