title: Gated Delta Networks: Improving Mamba2 with Delta Rule
summary: Linear Transformers have gained attention as efficient alternatives to
  standard Transformers, but their performance in retrieval and long-context
  tasks has been limited. To address these limitations, recent work has explored
  two distinct mechanisms: gating for adaptive memory control and the delta
  update rule for precise memory modifications. We observe that these mechanisms
  are complementary: gating enables rapid memory erasure while the delta rule
  facilitates targeted updates. Building on this insight, we introduce the gated
  delta rule and develop a parallel training algorithm optimized for modern
  hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
  existing models like Mamba2 and DeltaNet across multiple benchmarks, including
  language modeling, common-sense reasoning, in-context retrieval, length
  extrapolation, and long-context understanding. We further enhance performance
  by developing hybrid architectures that combine Gated DeltaNet layers with
  sliding window attention or Mamba2 layers, achieving both improved training
  efficiency and superior task performance.
paper_authors._id: 6758bd942c67b7cd7088bc21
paper_authors.name: Songlin Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-10T22:15:49.304Z
paper_authors._id: 65810ad10d738799ecbce2de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
paper_authors.fullname: Songlin Yang
paper_authors.type: user
paper_authors.user: sonta7
paper_authors._id: 6758bd942c67b7cd7088bc22
paper_authors.name: Jan Kautz
paper_authors._id: 6758bd942c67b7cd7088bc23
paper_authors.name: Ali Hatamizadeh
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-10T22:15:49.304Z
paper_authors._id: 64414b62603214724ebd2636
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
paper_authors.fullname: Ali
paper_authors.type: user
paper_authors.user: ahatamiz
paper_title: Gated Delta Networks: Improving Mamba2 with Delta Rule
paper_summary: Linear Transformers have gained attention as efficient alternatives to
  standard Transformers, but their performance in retrieval and long-context
  tasks has been limited. To address these limitations, recent work has explored
  two distinct mechanisms: gating for adaptive memory control and the delta
  update rule for precise memory modifications. We observe that these mechanisms
  are complementary: gating enables rapid memory erasure while the delta rule
  facilitates targeted updates. Building on this insight, we introduce the gated
  delta rule and develop a parallel training algorithm optimized for modern
  hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
  existing models like Mamba2 and DeltaNet across multiple benchmarks, including
  language modeling, common-sense reasoning, in-context retrieval, length
  extrapolation, and long-context understanding. We further enhance performance
  by developing hybrid architectures that combine Gated DeltaNet layers with
  sliding window attention or Mamba2 layers, achieving both improved training
  efficiency and superior task performance.
paper_ai_keywords: Art Transformers standard Transformers delta update rule gating memory erasure targeted updates Gated Delta rule parallel training algorithm modern hardware recall architecture Gated OptimizationNet Mamba2 DeltaNet sliding window attention length extrapolation long-context understanding hybrid architectures training efficiency task performance