The attention now spans beyond 20 turns. We tested the model ability to remember the system arguments
trough a human evaluation. The arguments (e.g. hobbies, persona) are defined during the first message, and
then from turn 2 to 20. We explicitly asked the model to refer to them (e.g. “What is your favorite hobby?”,
“What is your name?”), to measure the multi-turn memory ability of Llama 2-Chat. We report the results
in Table 30. Equipped with GAtt, Llama 2-Chat maintains 100% accuracy, always referring to the defined
attribute, and so, up to 20 turns (we did not extend the human evaluation more, and all the examples had
less than 4048 tokens in total over the turns). As a comparison, Llama 2-Chat without GAtt can not anymore
refer to the attributes after only few turns: from 100% at turn t+1, to 10% at turn t+3 and then 0%.
Llama2