ting preference data. In this setup we only add examples
where the candidate response was evaluated to give a perfect score of rn
i = 5.
While we report the results of both approaches in our experiments, we find that learning
from preference pairs gives superior performance, and hence recommend that approach.
2.4
Overall Self-Alignment Algorithm
Iterative Training
Our overall procedure trains a series of models M1, . . . , MT where
each successive model t uses augmented training data created by the t −1th m