% end in correct solution
% correct steps

85.1
58.6

13.2
74.1

14.2
73.1

phase 1

phase 2

combined

Table 3: Distribution of positive/negative steps/solutions.

Some of our phase 2 questions are intended for quality control. For a quality
control question, researchers mark which steps are reasonable to label as in-
correct. Then we assess that labelers are able to consistently mark those steps
as incorrect. Prior to starting on phase 2, we required all labelers to label 30
quality control questions. This served as a screening test, and we only admitted
labelers that agreed with our gold labels at least 75% of the time.

We then designated 10-20 problems per generation as additional quality
control questions, and we randomly served them to labelers as they worked

17
Let’s Verify Step by Step