3.2.2 Reward Modeling
The reward model takes a model response and its corresponding prompt (including contexts from previous
turns) as inputs and outputs a scalar score to indicate the quality (e.g., helpfulness and safety) of the model
generation. Leveraging such response scores as rewards, we can optimize Llama 2-Chat during RLHF for
better human preference alignment and improved helpfulness and safety.
Others have found that helpfulness and safety sometimes trade off (Bai et al., 2022a), which can make it
challenging for a single reward model to perform well on both. To address this, we train two separate reward
models, one optimized for helpfulness (referred to as Helpfulness RM) and another for safety (Safety RM).
We initialize our reward models from pretrained chat model checkpoints, as it ensures that both models
benefit from knowledge acquired in pretraining. In short, the reward model “knows” what the chat model

10

Avg. # Turns
per Dialogue

Avg. # Tokens
per Example
Llama2