6.2 Alignment Impact

Process supervision has several advantages over outcome supervision related
to AI alignment. Process supervision is more likely to produce interpretable
reasoning, since it encourages models to follow a process endorsed by humans.
Process supervision is also inherently safer: it directly rewards an aligned chain-
of-thought rather than relying on outcomes as a proxy for aligned behavior
(Stuhlm¨uller and Byun, 2022). In contrast, outcome supervision is harder to
scrutinize, and the preferences conveyed are less precise.
In the worst case,
the use of outcomes as an imperfect proxy could lead to models that become
misaligned after learning to exploit the reward signal (Uesato et al., 2022; Cotra,
2022; Everitt et al., 2017).
Let’s Verify Step by Step