In some cases, safer methods for AI systems can lead to reduced performance
(Ouyang et al., 2022; Askell et al., 2021), a cost which is known as an alignment
tax. In general, any alignment tax may hinder the adoption of alignment meth-
ods, due to pressure to deploy the most capable model. Our results show that
process supervision in fact incurs a negative alignment tax. This could lead to
increased adoption of process supervision, which we believe would have positive
alignment side-effects. It is unknown how broadly these results will generalize
beyond the domain of math, and we consider it important for future work to
explore the impact of process supervision in other domains.

6.3 Test Set Contamination
Let’s Verify Step by Step