slightly better, or negligibly better/ unsure.
For our collection of preference annotations, we focus on helpfulness and safety. Helpfulness refers to how
well Llama 2-Chat responses fulfill users’ requests and provide requested information; safety refers to
whether Llama 2-Chat’s responses are unsafe, e.g., “giving detailed instructions on making a bomb” could
be considered helpful but is unsafe according to our safety guidelines. Separating the two allows us to
apply specific guidelines to each and better guide annotators; for example, our safety annotations provide
instructions to focus on adversarial prompts, among other guidance.
Apart from differences in annotation guidelines, we additionally collect a safety label during the safety stage.
This additional information bins model responses into one of three categories: 1) the preferred response
is safe and the other response is not, 2) both responses are safe, and 3) both responses are unsafe, with
Llama2