2,043,904 tokens (Figure 1). RMT holds up surprisingly well on such long sequences, with Detect &
memorize being the easiest and Reasoning task the most complex.
Scaling Transformer to 1M tokens and beyond with RMT