3 Memorization Tasks

To test memorization abilities, we constructed synthetic datasets that require memorization of simple
facts and basic reasoning. The task input consists of one or several facts and a question that can be
answered only by using all of these facts. To increase the task difﬁculty, we added natural language
text unrelated to the questions or answers. This text acts as noise, so the model’s task is to separate
facts from irrelevant text and use them to answer the questions. The task is formulated as a 6-class
classiﬁcation, with each class representing a separate answer option.
Facts are generated using the bAbI dataset (Weston et al., 2016), while the background text is sourced
from questions in the QuALITY (Pang et al., 2022) long QA dataset.

Background text: ... He was a big man, broad-shouldered and still thin-waisted.
Eddie found it easy to believe the stories he had heard about his father ...

3.1 Fact Memorization
Scaling Transformer to 1M tokens and beyond with RMT