Our work revolves around the concept of memory in neural architectures. Memory has been a
recurrent theme in neural network research, dating back to early works (McCulloch and Pitts, 1943;
Stephen, 1956) and signiﬁcantly advancing in the 1990s with the introduction of the Backpropagation
Through Time learning algorithm (Werbos, 1990) and Long-Short Term Memory (LSTM) neural
architecture (Hochreiter and Schmidhuber, 1997). Contemporary memory-augmented neural net-
works (MANNs) typically utilize some form of recurrent external memory separate from the model’s
parameters. Neural Turing Machines (NTMs) (Graves et al., 2014) and Memory Networks (Weston
et al., 2015) are equipped with storage for vector representations accessible through an attention
mechanism. Memory Networks (Weston et al., 2015; Sukhbaatar et al., 2015) were designed to
enable reasoning through sequential attention over memory content.
Scaling Transformer to 1M tokens and beyond with RMT