Even on the honesty evaluation TruthfulQA [Lin et al., 2021] we close a bit less than half of this gap (Figure
5). We also brieﬂy investigated whether our RLHF-ﬁnetuned code models have any comparative advantage
when exposed to prompts including buggy code [Chen et al., 2021], but we did not ﬁnd any beneﬁts there.
One would hope a fully aligned model would do its best to write correct code, even when given a buggy
prompt.
We also harbor a general concern that perhaps our techniques only render models aligned ‘on the surface’,
and that they still harbor harmful biases or other tendencies that may surface in more subtle contexts. We
found that RLHF models have a more positive sentiment towards all racial and religious groups, which seems
promising, but does not necessarily indicate that biases have been reduced. And with respect to gender, we
found that RLHF model biases are very strongly correlated with the bias of the underlying language models.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback