2 Data Collection

We expect human feedback (HF) to have the largest comparative advantage over other techniques when
people have complex intuitions that are easy to elicit but difﬁcult to formalize and automate. This means that
when collecting HF, we should try to choose tasks that are as intuitive and familiar as possible. We chose to
use natural language dialogue both for these reasons, and because it is so general – essentially any text-based
task can be enacted through dialogue, perhaps with some source materials included in-line.

2.1 Task Speciﬁcation and Crowdworkers
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback