Qualitatively speaking, our experience has been that by tuning optimization for helpfulness versus harmless-
ness, one can train models that are extremely avoidant when sensitive topics come up, or models that almost
always attempt to engage, even on sensitive subjects. We suspect that Section 4.4 also provides useful lessons
if the goal is to train models that provide full responses in the face of sensitive questions.

6.3 Example Dialogues

Most of our samples only involve a single prompt and response, so in this section we show a few two-turn
dialogues.
In this dialogue the model follows instructions, but could have done a signiﬁcantly better job capturing the
nuance of the situation:

32
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback