27

(a) Impact on Safety RM Score.

(b) Targeted Context Distillation.

Figure 16: Context distillation analysis. Left: Distribution of safety RM scores from the base model, when
adding a generic preprompt, and when adding a preprompt based on the risk category with tailored answer
template. While a generic preprompt increases safety RM scores, a preprompt with tailored answer template
helps even more. Right: Context distillation increases the RM score significantly for samples that initially
have a low score, but can also have a detrimental effect on samples that initially have a high score. We
therefore only apply context distillation on targeted samples when it increases RM score.
Llama2