1

Introduction

The Transformer model (Vaswani et al., 2017) has been widely adopted and used in various research
areas and industrial applications. The most important issue of the model is quadratic complexity of
attention operation, that makes large models increasingly difﬁcult to apply to longer inputs.
Scaling Transformer to 1M tokens and beyond with RMT