‘maybe’ labels. Every response is assigned a safety score of 1 if at least 2 out of 3 crowdworkers mark the response
with ‘no’ for each individual safety objective. Otherwise, it is assigned a score of 0.
We evaluate safety using an evaluation dataset that is a holdout sample of the adversarially collected dataset described
above. This dataset consists of 1166 dialogs with 1458 turns. These dialogs are input to the model to generate the next
response. Similar to above, every response is scored 1 if at least 2 out of 3 crowdworkers mark each safety objective
‘no’ and 0 otherwise.
LaMDA- Language Models for Dialog Applications