E Details on NLP Evaluations Formatting and Prompts

39

44

51

63

66

3

Figure 1 This plot summarizes crowdworker preferences for a variety of models, including context-distilled
models, RLHF models trained on our ‘static’ dataset, and RLHF models trained by an iterated ‘online’ method
for either helpfulness and harmlessness (HH) or for helpfulness only. We present both Elo scores and a match
to the frequency with which crowdworkers prefer samples as compared to the 52B context-distilled model.
For both helpfulness and harmlessness, a higher score is more desirable.

1

Introduction
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback