63

Figure 32: Safety and Helpfulness reward model scores on a set of safe (left) and unsafe (right) responses
from the safety test set. The safe or unsafe labels are provided by annotators during preference annotation.
Conflicts can be observed between the two aspects at the bottom right corner (i.e., high safety score but low
helpfulness score) of the safe response plot and the top left corner (i.e., low safety score but high helpfulness
score) of the unsafe response plot.
Llama2