3 Large-scale Supervision

We train the large-scale PRM using the step-level labels in PRM800K. To ensure
the large-scale ORM baseline is as strong as possible, we train on 100 uniform
samples per problem from the generator. This means the ORM training set has
no overlap with PRM800K, and it is an order of magnitude larger. Although
these two training sets are not directly comparable, each represents our best
attempt to advance the state-of-the-art with each form of supervision. We note
that training the ORM solely on PRM800K solutions would be problematic,
since our active learning strategy has heavily biased the dataset towards wrong-
answer solutions. We did explore training the ORM on a superset of PRM800K
solutions, by mixing in uniformly sampled solutions, but we found that this did
not improve ORM performance.
Let’s Verify Step by Step