Alignment with Human Values Has Many Beneﬁts and Essentially No Cost to Performance

• Smaller models experience severe ‘alignment taxes’ – their performance on a wide variety of eval-
uations declines after RLHF training. However, we ﬁnd a variety of alignment bonuses, with our
13B and 52B5 RLHF-trained models performing better at zero-shot NLP evaluations, and the same
at few-shot evaluations.

• Natural language RLHF training for HH can be applied to models that have been ﬁrst ﬁnetuned
on code, and it improves their programming ability on evaluations (presumably by improving
general-purpose instruction following). We also ﬁnd that mixing preference model training for HH
with the specialized skill of summarization [Stiennon et al., 2020] incurs no degradation in perfor-
mance in either HH or summarization. So there is no reason not to combine alignment training with
more speciﬁc, valuable skills.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback