title: Breaking the Attention Bottleneck
summary: Attention-based transformers have become the standard architecture in many
  deep learning fields, primarily due to their ability to model long-range
  dependencies and handle variable-length input sequences. However, the attention
  mechanism with its quadratic complexity is a significant bottleneck in the
  transformer architecture. This algorithm is only uni-directional in the decoder
  and converges to a static pattern in over-parametrized decoder-only models. I
  address this issue by developing a generative function as attention or
  activation replacement. It still has the auto-regressive character by comparing
  each token with the previous one. In my test setting with nanoGPT this yields a
  smaller loss while having a smaller model. The loss further drops by
  incorporating an average context vector. This concept of attention replacement
  is distributed under the GNU AGPL v3 license at
  https://gitlab.com/Bachstelze/causal_generation.
paper_authors._id: 6671219645b689bdb3f8ef60
paper_authors.name: Kalle Hilsenbek
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:17:34.044Z
paper_authors._id: 62d1ddfac58f969c1528f1b5
paper_authors.avatarUrl: /avatars/75c372a831cde3c7c6dce3bc875488a7.svg
paper_authors.fullname: Kalle Hilsenbek
paper_authors.type: user
paper_authors.user: Bachstelze
paper_title: Breaking the Attention Bottleneck
paper_summary: Attention-based transformers have become the standard architecture in many
  deep learning fields, primarily due to their ability to model long-range
  dependencies and handle variable-length input sequences. However, the attention
  mechanism with its quadratic complexity is a significant bottleneck in the
  transformer architecture. This algorithm is only uni-directional in the decoder
  and converges to a static pattern in over-parametrized decoder-only models. I
  address this issue by developing a generative function as attention or
  activation replacement. It still has the auto-regressive character by comparing
  each token with the previous one. In my test setting with nanoGPT this yields a
  smaller loss while having a smaller model. The loss further drops by
  incorporating an average context vector. This concept of attention replacement
  is distributed under the GNU AGPL v3 license at
  https://gitlab.com/Bachstelze/causal_generation.
paper_ai_keywords: attention-based transformers long-range dependencies quadratic complexity auto-regressive character generative function average context vector attention replacement