level sophistication. In any case, this is just one speculative possibility for broadening participation in dataset
creation.
Our research has beneﬁted from publicly available research datasets and evaluations relevant to aligning AI
with human values [Stiennon et al., 2020, Hendrycks et al., 2021a], and we plan to release our preference
modeling data for others to use in their research. Unfortunately, this does not seem to be a standard practice
among alignment researchers, as evidenced by some recent work. While we agree that LLMs themselves can
be used for harm, it seems that no such argument can be made for alignment data.
It’s extremely important to enable collaboration and reproducibility for alignment and safety research. As
AI systems become more powerful and more widely deployed, the cost of mistakes and misunderstandings
may grow immensely. We believe that the only way to convincingly address potential safety failures from
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback