Lranking = −log(σ(rθ(x, yc) − rθ(x, yr)))

(1)
where rθ(x, y) is the scalar score output for prompt x and completion y with model weights θ. yc is the
preferred response that annotators choose and yr is the rejected counterpart.
Built on top of this binary ranking loss, we further modify it separately for better helpfulness and safety
reward models as follows. Given that our preference ratings is decomposed as a scale of four points (e.g.,
significantly better), as presented in Section 3.2.1, it can be useful to leverage this information to explicitly
teach the reward model to assign more discrepant scores to the generations that have more differences. To
do so, we further add a margin component in the loss:

Lranking = −log(σ(rθ(x, yc) − rθ(x, yr) − m(r)))
Llama2