querieskeysDetect fact 1 No fact in inputUse memory Z...write to memoryretain memoryDetect fact 2write to memoryanswer the questionsimilar approach to Memformer, MART (Lei et al., 2020) adopts memory update rules analogous to
LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Cho et al., 2014). FeedBack Transformer (Fan
et al., 2020) implements full recurrence beyond the segment level.
A drawback of most existing recurrent methods is the need for architectural modiﬁcations that
complicate their application to various pre-trained models. In contrast, the Recurrent Memory
Transformer can be built upon any model that uses a common supported interface.
Some approaches redesign the self-attention mechanism to reduce computational complexity while
minimizing input coverage loss. Star-Transformer (Guo et al., 2019), Longformer (Beltagy et al.,
2020), GMAT (Gupta and Berant, 2020), Extended Transformer Construction (ETC) (Ainslie et al.,
Scaling Transformer to 1M tokens and beyond with RMT