A.3.4 Ablation on Ranking Loss with Safety Auxiliary Loss for Reward Modeling
We ablated the impact of the safety auxiliary loss with results on the Meta Safety test set shown in Table 29.
As expected, The customized loss improves the recall of unsafe responses when we use a reward score of 0.5
as the threshold (negative before Sigmoid) and thus offers a better safety reward signal for RLHF. Teaching
the model to discriminate between safe and unsafe model generations also improves model accuracy on three
subcategories.

52

Figure 25: Distribution of human preference data rating over batches. Over time, the share of samples
with an unsure or negligibly better rating become larger with better performing Llama 2-Chat trained and
available for preference data annotation.

Safe Chosen
Unsafe Rejected

Safe Chosen
Safe Rejected

Unsafe Chosen
Unsafe Rejected

Unsafe Response
Llama2