D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
P. Christiano, and G. Irving. Fine-tuning language models from human pref-
erences. arXiv preprint arXiv:1909.08593, 2019.

15

A MathMix

Similar to Lewkowycz et al. (2022) we construct a large-scale dataset of high-
quality math-relevant tokens for use in a lightweight pretraining stage, before
finetuning on comparably smaller datasets like MATH and PRM800K. This
dataset, which we call MathMix, has two main differences compared to the one
used to train Minerva. First, it is smaller and more aggressively filtered to high-
quality math problem-solving content, and second, it does not explicitly mix in
general language data.
Let’s Verify Step by Step