Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. In
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1315–1325, Min-
neapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1133.
URL https://aclanthology.org/N19-1133.

Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint

arXiv:2006.03274, 2020.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735–1780,
November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/
neco.1997.9.8.1735.
Scaling Transformer to 1M tokens and beyond with RMT