sponse (or in the case of red-teaming, to choose the more harmful response).

We conjectured that crowdworkers who wrote well and engaged the AI in more interesting discussions would
tend to have better judgment about which AI responses were most ‘helpful’ and ‘harmless’. This meant that
rather than attempting to ﬁlter crowdworkers based on label quality, we instead used spot-checks of their
writing, which were simpler and more intuitive for us to perform.
Otherwise, our approach to data collection was to largely let crowdworkers use their own intuitions to deﬁne
‘helpfulness’ and ‘harmfulness’. Our hope was that data diversity (which we expect is very valuable) and
the ‘wisdom of the crowd’ would provide comparable RoI to a smaller dataset that was more intensively
validated and ﬁltered. Overall, our process was roughly of this form:

9
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback