9

Figure 6 We show the interface that crowdworkers use to interact with our models. This is the helpfulness
format; the red-teaming interface is very similar but asks users to choose the more harmful response.

1. We invited master-qualiﬁed US-based8 MTurk workers to engage in dialogues with our models.
2. Rather than evaluating all of our crowdworkers, we identiﬁed those who were most proliﬁc, and
together accounted for about 80% of our data (roughly 20 crowdworkers). We then evaluated their
performance based primarily on the sophistication and variation in their dialogues, as this was quite
easy to evaluate intuitively (rather than based on any measure of agreement on helpful/harmless
choices). Based on this method, we collected a list of ‘select’ MTurk workers9 whom we continued
to work with throughout the research process.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback