7 Discussion

The problem of long inputs in Transformers has been extensively researched since the popularization
of this architecture. In this work, we demonstrate that applying Transformers to long texts does not
necessarily require large amounts of memory. By employing a recurrent approach and memory, the
quadratic complexity can be reduced to linear. Furthermore, models trained on sufﬁciently large
inputs can extrapolate their abilities to texts orders of magnitude longer.
Synthetic tasks explored in this study serve as the ﬁrst milestone for enabling RMT to generalize to
tasks with unseen properties, including language modelling. In our future work, we aim to tailor
the recurrent memory approach to the most commonly used Transformers to improve their effective
context size.

References
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Etc:

Encoding long and structured data in transformers, 2020.
Scaling Transformer to 1M tokens and beyond with RMT