unbounded memory, 2015.

Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio. Dynamic neural turing machine with

soft and hard addressing schemes. arXiv preprint arXiv:1607.00036, 2016.

Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory augmented neural networks with wormhole

connections. arXiv preprint arXiv:1701.08718, 2017.

Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang.
LongT5: Efﬁcient text-to-text transformer for long sequences. In Findings of the Association for Com-
putational Linguistics: NAACL 2022, pages 724–736, Seattle, United States, July 2022. Association for
Computational Linguistics. doi: 10.18653/v1/2022.ﬁndings-naacl.55. URL https://aclanthology.org/
2022.findings-naacl.55.
Scaling Transformer to 1M tokens and beyond with RMT