The idea is that the golden labels are treated as a source of truth, with labelers incentivized to modify their
behavior to better match the golden labels, and novice labelers incentivized to match the behavior of more
experienced labelers.
These sorts of methods weren’t easily adapted to our data collection setup. Open-ended conversations allowed
us to collect a richer dataset, but introduced a number of hard-to-control variables, resulting in noisy data
quality metrics.
We did try having crowdworkers review each other’s conversations, by providing ratings for each model
response comparison and rating the overall quality of the conversation. But we found that author-rater agree-
ment wasn’t a good guide for assessing overall conversation quality. In broad strokes, conversation quality
depends on a) conversation topic, b) human writing quality, c) model quality. And we found that, for instance,
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback