as conversations got more sophisticated, deciding between model responses got more difﬁcult. As a result,
authors that more frequently discussed difﬁcult topics would often get lower agreement scores. And because
we were frequently updating our models, model response comparisons were a moving target.
Instead of directly assessing our crowdworkers based on these kinds of reviewer-based metrics, we considered
using their performance as reviewers as a stand-in for their performance as conversation authors. But when we
compared these metrics against our own spot-checks of conversation author quality, we found poor agreement.
We expect to be able to work around these problems and come up with better methods in the future. But it’s
worth noting that were able to achieve our results without sophisticated data quality controls.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback