Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric
Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language
processing. In Proceedings of the 2020 conference on empirical methods in natural language processing:
system demonstrations, pages 38–45, 2020.

Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-
augmented transformer for sequence modeling. In Findings of the Association for Computational Linguistics:
AACL-IJCNLP 2022, pages 308–318, Online only, November 2022a. Association for Computational Linguis-
tics. URL https://aclanthology.org/2022.findings-aacl.29.

Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In
International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?
id=TrjbxzRcnf-.
Scaling Transformer to 1M tokens and beyond with RMT