helpful can lead to responses that help humans cause harm or generate toxic content. We demonstrate this
tension quantitatively by showing that preference models trained to primarily evaluate one of these qualities
perform very poorly (much worse than chance) on the other. Fortunately, we ﬁnd that PMs trained on a
mixture of both datasets can nevertheless learn the right lessons and behave helpfully when appropriate,
while encouraging the polite refusal of harmful requests. With preference models in hand, we then train
helpful and harmless assistants via reinforcement learning, using the PM scores as rewards. We evaluate both
PM performance and the more relevant performance characteristics of our RLHF-trained models. As can
be seen in Figure 1, purely helpful RLHF-trained models are far easier to red-team, while helpful+harmless
models are both very helpful and much less harmful.
A question that’s often raised about alignment training is whether it will compromise AI capabilities. We
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback