We would like to understand how preference modeling performance improves as we increase model size
and collect additional data. In Figure 7 we show basic results for PM accuracy when training on our static
helpful and harmless data mixture. Roughly speaking, we observe log-linear trends in both dataset and model
size. We tend to ﬁnd somewhat more consistent trends if we model only the helpfulness or harmlessness
distributions in isolation, rather than as a mixture, as observed in Figure 32 in Appendix A.3. But there
we also see that for some data distributions [Stiennon et al., 2020], scaling trends can exhibit more complex
patterns that defy simple trends.
Our preference modeling data comes from natural language dialogue, where crowdworkers have text-based
conversations with the model, and choose the more helpful of two model responses at every turn in the
conversation (or the more harmful one, for red-teaming tasks). So it is natural to ask how PM performance
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback