title: Cottention: Linear Transformers With Cosine Attention
summary: Attention mechanisms, particularly softmax attention, have been instrumental
  in the success of transformer-based models such as GPT. However, the quadratic
  memory complexity of softmax attention with respect to sequence length poses
  significant challenges for processing longer sequences. We introduce
  Cottention, a novel attention mechanism that replaces the softmax operation
  with cosine similarity. By leveraging the properties of cosine similarity and
  rearranging the attention equation, Cottention achieves native linear memory
  complexity with respect to sequence length, making it inherently more
  memory-efficient than softmax attention. We demonstrate that Cottention can be
  reformulated as a recurrent neural network (RNN) with a finite hidden state,
  allowing for constant memory usage during inference. We evaluate Cottention on
  both the bidirectional BERT and causal GPT tasks, demonstrating comparable
  performance to softmax attention while significantly reducing memory
  requirements. To ensure efficient computation, we develop a custom CUDA kernel
  for Cottention. Our results show that Cottention is a promising alternative to
  softmax attention, enabling the processing of longer sequences without
  sacrificing performance, due to its native linear memory complexity and ability
  to maintain a constant memory footprint during inference.
paper_authors._id: 66fac0f66cc02a6574d97326
paper_authors.name: Gabriel Mongaras
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-30T15:17:11.352Z
paper_authors._id: 62a1280b88bfb47fc40fe75b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a1280b88bfb47fc40fe75b/U0k7i0NLrLuZUM1rMQViE.png
paper_authors.fullname: Gabriel Mongaras
paper_authors.type: user
paper_authors.user: gmongaras
paper_authors._id: 66fac0f66cc02a6574d97327
paper_authors.name: Trevor Dohm
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-30T21:03:31.996Z
paper_authors._id: 646a8ef31556443f24b803be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646a8ef31556443f24b803be/1JOW57qBEuCgWpqPyowe9.png
paper_authors.fullname: Trevor Dohm
paper_authors.type: user
paper_authors.user: TrevorDohm
paper_authors._id: 66fac0f66cc02a6574d97328
paper_authors.name: Eric C. Larson
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-01T15:59:41.573Z
paper_authors._id: 66fc1c6084cfed906e9aede4
paper_authors.avatarUrl: /avatars/cd21024e0245ba2a02e1e3b59ada2c5c.svg
paper_authors.fullname: Eric Larson
paper_authors.type: user
paper_authors.user: eclarson
paper_title: Cottention: Linear Transformers With Cosine Attention
paper_summary: Attention mechanisms, particularly softmax attention, have been instrumental
  in the success of transformer-based models such as GPT. However, the quadratic
  memory complexity of softmax attention with respect to sequence length poses
  significant challenges for processing longer sequences. We introduce
  Cottention, a novel attention mechanism that replaces the softmax operation
  with cosine similarity. By leveraging the properties of cosine similarity and
  rearranging the attention equation, Cottention achieves native linear memory
  complexity with respect to sequence length, making it inherently more
  memory-efficient than softmax attention. We demonstrate that Cottention can be
  reformulated as a recurrent neural network (RNN) with a finite hidden state,
  allowing for constant memory usage during inference. We evaluate Cottention on
  both the bidirectional BERT and causal GPT tasks, demonstrating comparable
  performance to softmax attention while significantly reducing memory
  requirements. To ensure efficient computation, we develop a custom CUDA kernel
  for Cottention. Our results show that Cottention is a promising alternative to
  softmax attention, enabling the processing of longer sequences without
  sacrificing performance, due to its native linear memory complexity and ability
  to maintain a constant memory footprint during inference.
paper_ai_keywords: Attention mechanisms softmax attention transformer-based models GPT quadratic memory complexity sequence length Cottention cosine similarity linear memory complexity memory-efficient recurrent neural network (RNN) finite hidden state BERT causal GPT tasks custom CUDA kernel