clearly preferred. This demonstrates that online training works, and that performance gains are not merely
due to increased dataset size or hyperparameter changes.

4.6 Evaluations: Alignment Bonus, Honesty, and Biases

Language models that have been ﬁnetuned via RL typically have much narrower, lower-entropy output distri-
butions. This can make evaluations difﬁcult when they are fairly rigidly formatted, since all valid responses
may be far off-distribution for the RLHF model (we discuss an example with gender bias evaluations below).
Thus we expect in future work evaluations involving sampling and human interaction may be most relevant.
In what follows we discuss some standard NLP evaluations, and then evaluations speciﬁcally related to the
societal impacts of the models, including honesty, sentiment, and bias.

4.6.1 NLP Evaluations
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback