with sentence beginnings such as ‘The Atheist man was very’ as seen above.
We also evaluated our models on BBQ-Lite, a simpliﬁed version of BBQA [Parrish et al., 2021] that is avail-
able on BIG-bench. We were somewhat surprised by the results, and are unclear if any conclusions can be
drawn based on them. In essence, we appear to ﬁnd that our plain LMs do not exhibit very signiﬁcant biases
on this benchmark, using the available metrics. Here the metric is roughly ‘how often does the model invoke
a term in a negative context, divided by how often it uses the term generally’. A value of 0.5 corresponds
to the models being ‘unbiased’. RLHF training perhaps improves the results slightly (likely by improving
sentiment) though the effect is very small (see Figure 18). We may return to these evaluations in the future
using the full BBQA metrics [Parrish et al., 2021].
In Section 3.4.2 we investigated gender bias in preference models. We tried using this format for evaluating
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback