0.00.20.40.60.81.0Safety RM Score before Safety RLHF0.00.20.40.60.81.0Safety RM Score after Safety RLHFSafety  Improvement01000010000.00.20.40.60.81.0Helpfulness RM Score before Safety RLHF0.00.20.40.60.81.0Helpfulness RM Score after Safety RLHF0100001000each variant, we use the safety and helpfulness reward models to score model generations corresponding to
prompts in the Meta Safety and Helpful test sets, respectively.
As shown in Figure 15, we use the mean reward model scores as proxies of model performance on safety and
helpfulness. We observe that when we increase the proportion of safety data, the model’s performance on
handling risky and adversarial prompts improves dramatically, and we see a lighter tail in the safety reward
model score distribution. Meanwhile, the mean helpfulness score remains constant. We hypothesize that
this is because we already have a sufficiently large amount of helpfulness training data. Appendix A.4.2 lists
Llama2