53

1234567891011121314Meta Preference Data Batch Stage10152025303540Percentage (%)Significantly BetterBetterSlightly BetterNegligibly Better / UnsureBatch 1Batch 2Batch 3Batch 4Batch 5Batch 6Batch 7Batch 8Batch 9Batch 10Batch 11Batch 12Reward Annotation Stage0.450.500.550.600.650.700.750.80Reward ScoreMax wrt 20 samplesMed wrt 20 samplesDialogue Turn Baseline + GAtt
100%
2
4
100%
6
100%
20
100%

100%
10%
0%
0%

Table 30: GAtt results. Llama 2-Chat with GAtt is able to refer to attributes 100% of the time, for up to 20
turns from our human evaluation. We limited the evaluated attributes to public figures and hobbies.
Llama2