Finally, we investigate the impact of active learning. We train a small-scale
reward model, PRMselector, on a single sample from each problem, and we use
this model to score 1000 samples per problem. To train each of our larger re-
ward models, we select N samples per problem such that 80% are the most
convincing (according to PRMselector) wrong-answer samples, and 20% are the
most convincing samples that remain (right- or wrong-answer). We score the
selected samples with PRMlarge and train on those scores. This process ensures
that all samples are relatively convincing under PRMselector, that a large frac-
tion are known to contain at least one mistake, and that our overall dataset
is not too heavily biased toward wrong-answer solutions. Performance of this
data labelling scheme is shown in Figure 4a. By comparing the slopes of the
line of best fit with and without active learning, we estimate that this form
Let’s Verify Step by Step