title: TransformerFAM: Feedback attention is working memory
summary: While Transformers have revolutionized deep learning, their quadratic
  attention complexity hinders their ability to process infinitely long inputs.
  We propose Feedback Attention Memory (FAM), a novel Transformer architecture
  that leverages a feedback loop to enable the network to attend to its own
  latent representations. This design fosters the emergence of working memory
  within the Transformer, allowing it to process indefinitely long sequences.
  TransformerFAM requires no additional weights, enabling seamless integration
  with pre-trained models. Our experiments show that TransformerFAM significantly
  improves Transformer performance on long-context tasks across various model
  sizes (1B, 8B, and 24B). These results showcase the potential to empower Large
  Language Models (LLMs) to process sequences of unlimited length.
paper_authors._id: 661df566816b6d879752a32d
paper_authors.name: Dongseong Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:39:15.711Z
paper_authors._id: 631523ea27d020853f673cba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631523ea27d020853f673cba/XF7l5kl2npvjZe8A7BrYD.jpeg
paper_authors.fullname: Dongseong Hwang
paper_authors.type: user
paper_authors.user: dongseong
paper_authors._id: 661df566816b6d879752a32e
paper_authors.name: Weiran Wang
paper_authors._id: 661df566816b6d879752a32f
paper_authors.name: Zhuoyuan Huo
paper_authors._id: 661df566816b6d879752a330
paper_authors.name: Khe Chai Sim
paper_authors._id: 661df566816b6d879752a331
paper_authors.name: Pedro Moreno Mengibar
paper_title: TransformerFAM: Feedback attention is working memory
paper_summary: While Transformers have revolutionized deep learning, their quadratic
  attention complexity hinders their ability to process infinitely long inputs.
  We propose Feedback Attention Memory (FAM), a novel Transformer architecture
  that leverages a feedback loop to enable the network to attend to its own
  latent representations. This design fosters the emergence of working memory
  within the Transformer, allowing it to process indefinitely long sequences.
  TransformerFAM requires no additional weights, enabling seamless integration
  with pre-trained models. Our experiments show that TransformerFAM significantly
  improves Transformer performance on long-context tasks across various model
  sizes (1B, 8B, and 24B). These results showcase the potential to empower Large
  Language Models (LLMs) to process sequences of unlimited length.
paper_ai_keywords: Feedback Attention Memory (FAM) quadratic attention complexity latent representations working memory indefinite long sequences pre-trained models long-context tasks Large Language Models (LLMs)