arg max

π

R(g | p) = ˜Rc(g | p) − βDKL(πθ(g | p) ∥ π0(g | p))

(4)
contains a penalty term for diverging from the original policy π0. As was observed in other works (Stiennon
et al., 2020; Ouyang et al., 2022), we find this constraint is useful for training stability, and to reduce reward
hacking whereby we would achieve high scores from the reward model but low scores from human evaluation.
We define Rc to be a piecewise combination of the safety (Rs) and helpfulness (Rh) reward models. We
have tagged prompts in our dataset that might elicit potentially unsafe responses and prioritize the scores
from the safety model. The threshold of 0.15 is chosen for filtering unsafe responses, corresponding to a
precision of 0.89 and a recall of 0.55 evaluated on the Meta Safety test set. We also find it important to whiten
the final linear scores (shown here by reversing the sigmoid with the logit function) in order to increase
stability and balance properly with the KL penalty term (β) above.
Llama2