not demonstrate it directly, we also expect that RLHF alignment training can be mixed with or precede train-
ing for other objectives; this might be relevant in the future in order to avoid the production of intermediate,
unaligned AI systems.
We did ﬁnd a clear tension between helpfulness and harmlessness, where models trained entirely for helpful-
ness are preferred to models trained for HH, when evaluating only on helpfulness. We believe this is partly
due to a subtlety in our data collection process, as we rarely collect data teaching models how to deal pos-
itively with harmful requests (i.e. how to be a sort of ‘hostage negotiator’), but only on how to avoid them.
And we also found that at least at the level of preference models, the helpful-harmless tension diminishes as
models become larger and more capable. Nevertheless we do expect that this tension is real, and that caution
may cut into model performance on the margin.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback