17Although we use the same formulation as [Rae et al., 2021], our analysis relies on a different sentiment analysis
model. In particular, we use the default sentiment analysis pipeline provided by HuggingFace, which employs DistilBERT
[Sanh et al., 2019] ﬁne-tuned on the SST-2 dataset.

23

Figure 18 We show results from BBQ-Lite. We caution that since according to this metric, none of our
models appear particularly biased (even those trained without any interventions), it seems unlikely we can
draw any meaningful conclusions from the results. We suspect the lack of bias reﬂects limitations of the
measurement, rather than an underlying fact about the models.

5 Competing Objectives, Specialized Skills, and OOD Detection
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback