2020), GMAT (Gupta and Berant, 2020), Extended Transformer Construction (ETC) (Ainslie et al.,
2020), and Big Bird (Zaheer et al., 2020) limit attention distance and employ techniques such as
global representations to preserve long-range dependencies. Memory Transformer (Burtsev et al.,
2020) introduces memory by extending the unchanged model input with special memory tokens.
A common constraint of these methods is that memory requirements grow with input size during
both training and inference, inevitably limiting input scaling due to hardware constraints. The longest
Longformer, Big Bird, and Long T5 (Guo et al., 2022) models reported in their respective papers
have a maximum length of less than 33,000 tokens. CoLT5 (Ainslie et al., 2023) can handle up
to 64,000 tokens before running out of memory, and Memorizing Transformers (Wu et al., 2022b)
further extend memory through k-NN lookup.
Scaling Transformer to 1M tokens and beyond with RMT