TransformerFAM: Feedback attention is working memory
While Transformers have revolutionized deep learning, their quadratic
attention complexity hinders their ability to process infinitely long inputs.
We propose Feedback Attention Memory (FAM), a novel Transformer architecture
that leverages a feedback loop to enable the network to attend to its own
latent representations. This design fosters the emergence of working memory
within the Transformer, allowing it to process indefinitely long sequences.
TransformerFAM requires no additional weights, enabling seamless integration
with pre-trained models. Our experiments show that TransformerFAM significantly
improves Transformer performance on long-context tasks across various model
sizes (1B, 8B, and 24B). These results showcase the potential to empower Large
Language Models (LLMs) to process sequences of unlimited length.
661df566816b6d879752a32d
Dongseong Hwang
admin_assigned
2024-04-18T09:39:15.711Z
631523ea27d020853f673cba
https://cdn-avatars.huggingface.co/v1/production/uploads/631523ea27d020853f673cba/XF7l5kl2npvjZe8A7BrYD.jpeg
Dongseong Hwang
user
dongseong
661df566816b6d879752a32e
Weiran Wang
661df566816b6d879752a32f
Zhuoyuan Huo
661df566816b6d879752a330
Khe Chai Sim
661df566816b6d879752a331
Pedro Moreno Mengibar
TransformerFAM: Feedback attention is working memory
While Transformers have revolutionized deep learning, their quadratic
attention complexity hinders their ability to process infinitely long inputs.
We propose Feedback Attention Memory (FAM), a novel Transformer architecture
that leverages a feedback loop to enable the network to attend to its own
latent representations. This design fosters the emergence of working memory
within the Transformer, allowing it to process indefinitely long sequences.
TransformerFAM requires no additional weights, enabling seamless integration
with pre-trained models. Our experiments show that TransformerFAM significantly
improves Transformer performance on long-context tasks across various model
sizes (1B, 8B, and 24B). These results showcase the potential to empower Large
Language Models (LLMs) to process sequences of unlimited length.
Feedback Attention Memory (FAM) enhances Transformer architecture by enabling long-context processing without additional weights, significantly improving performance on large sequences across various model sizes.
Transformers Feedback Attention Memory FAM quadratic attention complexity working memory long-context tasks Large Language Models LLMs