Extended Mind Transformers
<|reference_start|>Extended Mind Transformers: Pre-trained language models demonstrate general intelligence and common sense, but long inputs quickly become a bottleneck for memorizing information at inference time. We resurface a simple method, Memorizing Transformers (Wu et al., 2022), that gives the model access to a bank of pre-computed memories. We show that it is possible to fix many of the shortcomings of the original method, such as the need for fine-tuning, by critically assessing how positional encodings should be updated for the keys and values retrieved. This intuitive method uses the model's own key/query system to select and attend to the most relevant memories at each generation step, rather than using external embeddings. We demonstrate the importance of external information being retrieved in a majority of decoder layers, contrary to previous work. We open source a new counterfactual long-range retrieval benchmark, and show that Extended Mind Transformers outperform today's state of the art by 6% on average.<|reference_end|>
@article{klett2024extended,
    title={Extended Mind Transformers},
    author={Phoebe Klett and Thomas Ahle},
    journal={arXiv preprint arXiv:2406.02332},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02332},
    primaryClass={cs.LG cs.CL}
}