24

Figure 14: Impact of safety RLHF measured by reward model score distributions. Left: safety reward
model scores of generations on the Meta Safety test set. The clustering of samples in the top left corner
suggests the improvements of model safety. Right: helpfulness reward model scores of generations on the
Meta Helpfulness test set.
Llama2