A histogram of these distributions by our ﬁnal, online HH preference model’s score can be seen in Figure 15
in Section 4.5. In what follows when we discuss the static or base+RS dataset, we will be referring to the
combination of the ﬁrst two components. Our ‘online’ RLHF models in Section 4.5 are trained on all three
components. Most of our results are based on the static dataset, as we conducted experiments and evaluations
with it while the online data collection was underway.
We analyze a few different splits of the static dataset – a standard split into 95/5 train/test data, and a 65/35
split that we use in order to obtain better statistics when evaluating preference model calibration on the test
set. We also consider a 50/50 split, where we train distinct PMs on the two halves of the dataset. This is used
to evaluate the robustness of RL training, as we then train an RL policy against one PM while evaluating the
rewards achieved by that policy as measured by the independent PM.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback