crowdworkers. However, readers might worry about two caveats: the online model was trained on a slightly
larger (about 20% larger) dataset, and the online model was trained with improved RLHF hyperparameters
(the online model was trained with a larger K, deﬁned in Appendix B.1, and its PM was trained with 2048
context instead of 1024), as compared to the earlier static RLHF training run.
To address both of these caveats, we performed a controlled experiment comparing two RLHF runs: one
trained with our base dataset (about 44k PM comparisons), and another trained on an even mixture of base,
RS, and online data whose total dataset size is the same as the base dataset15 (about 15k PM comparisons
from each). So for this experiment we trained two separate PMs on each dataset, and then trained a pair of
RLHF policies against these two PMs. Apart from the data difference, both runs used the same settings, and
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback