title: Forgetting Transformer: Softmax Attention with a Forget Gate
summary: An essential component of modern recurrent sequence models is the forget
  gate. While Transformers do not have an explicit recurrent form, we show that a
  forget gate can be naturally incorporated into Transformers by down-weighting
  the unnormalized attention scores in a data-dependent way. We name this
  attention mechanism the Forgetting Attention and the resulting model the
  Forgetting Transformer (FoX). We show that FoX outperforms the Transformer on
  long-context language modeling, length extrapolation, and short-context
  downstream tasks, while performing on par with the Transformer on long-context
  downstream tasks. Moreover, it is compatible with the FlashAttention algorithm
  and does not require any positional embeddings. Several analyses, including the
  needle-in-the-haystack test, show that FoX also retains the Transformer's
  superior long-context capabilities over recurrent sequence models such as
  Mamba-2, HGRN2, and DeltaNet. We also introduce a "Pro" block design that
  incorporates some common architectural components in recurrent sequence models
  and find it significantly improves the performance of both FoX and the
  Transformer. Our code is available at
  https://github.com/zhixuan-lin/forgetting-transformer.
paper_authors._id: 67cc697fa029f09af72cca01
paper_authors.name: Zhixuan Lin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-08T16:00:21.933Z
paper_authors._id: 6694cc1009326cb83f2d11bb
paper_authors.avatarUrl: /avatars/1ddaaed70a16ac475a9404848aef5d48.svg
paper_authors.fullname: Zhixuan Lin
paper_authors.type: user
paper_authors.user: zhixuan-lin
paper_authors._id: 67cc697fa029f09af72cca02
paper_authors.name: Evgenii Nikishin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:54:09.945Z
paper_authors._id: 64234eadd654afd6931a288b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UTU-XcO_ssKpIYr5MBujK.jpeg
paper_authors.fullname: Evgenii Nikishin
paper_authors.type: user
paper_authors.user: nikishin
paper_authors._id: 67cc697fa029f09af72cca03
paper_authors.name: Xu Owen He
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-08T16:00:01.392Z
paper_authors._id: 66906c4e37eadb9c577984d3
paper_authors.avatarUrl: /avatars/b81765472942fdf94c0ee885ca62df2d.svg
paper_authors.fullname: Owen He
paper_authors.type: user
paper_authors.user: littleowen
paper_authors._id: 67cc697fa029f09af72cca04
paper_authors.name: Aaron Courville
paper_title: Forgetting Transformer: Softmax Attention with a Forget Gate
paper_summary: An essential component of modern recurrent sequence models is the forget
  gate. While Transformers do not have an explicit recurrent form, we show that a
  forget gate can be naturally incorporated into Transformers by down-weighting
  the unnormalized attention scores in a data-dependent way. We name this
  attention mechanism the Forgetting Attention and the resulting model the
  Forgetting Transformer (FoX). We show that FoX outperforms the Transformer on
  long-context language modeling, length extrapolation, and short-context
  downstream tasks, while performing on par with the Transformer on long-context
  downstream tasks. Moreover, it is compatible with the FlashAttention algorithm
  and does not require any positional embeddings. Several analyses, including the
  needle-in-the-haystack test, show that FoX also retains the Transformer's
  superior long-context capabilities over recurrent sequence models such as
  Mamba-2, HGRN2, and DeltaNet. We also introduce a "Pro" block design that
  incorporates some common architectural components in recurrent sequence models
  and find it significantly improves the performance of both FoX and the
  Transformer. Our code is available at
  https://github.com/zhixuan-lin/forgetting-transformer.
paper_ai_keywords: recurrent sequence models forget gate Transformers Forgetting Attention Forgetting Transformer (FoX) long-context language modeling length extrapolation short-context downstream tasks long-context downstream tasks FlashAttention algorithm positional embeddings needle-in-the-haystack test long-context capabilities Mamba-2 HGRN2 DeltaNet Pro block design