Gated Delta Networks: Improving Mamba2 with Delta Rule
Linear Transformers have gained attention as efficient alternatives to
standard Transformers, but their performance in retrieval and long-context
tasks has been limited. To address these limitations, recent work has explored
two distinct mechanisms: gating for adaptive memory control and the delta
update rule for precise memory modifications. We observe that these mechanisms
are complementary: gating enables rapid memory erasure while the delta rule
facilitates targeted updates. Building on this insight, we introduce the gated
delta rule and develop a parallel training algorithm optimized for modern
hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
existing models like Mamba2 and DeltaNet across multiple benchmarks, including
language modeling, common-sense reasoning, in-context retrieval, length
extrapolation, and long-context understanding. We further enhance performance
by developing hybrid architectures that combine Gated DeltaNet layers with
sliding window attention or Mamba2 layers, achieving both improved training
efficiency and superior task performance.
6758bd942c67b7cd7088bc21
Songlin Yang
extracted_pending
2024-12-10T22:15:49.304Z
65810ad10d738799ecbce2de
https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
Songlin Yang
user
sonta7
6758bd942c67b7cd7088bc22
Jan Kautz
6758bd942c67b7cd7088bc23
Ali Hatamizadeh
extracted_pending
2024-12-10T22:15:49.304Z
64414b62603214724ebd2636
https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
Ali
user
ahatamiz
Gated Delta Networks: Improving Mamba2 with Delta Rule
Linear Transformers have gained attention as efficient alternatives to
standard Transformers, but their performance in retrieval and long-context
tasks has been limited. To address these limitations, recent work has explored
two distinct mechanisms: gating for adaptive memory control and the delta
update rule for precise memory modifications. We observe that these mechanisms
are complementary: gating enables rapid memory erasure while the delta rule
facilitates targeted updates. Building on this insight, we introduce the gated
delta rule and develop a parallel training algorithm optimized for modern
hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
existing models like Mamba2 and DeltaNet across multiple benchmarks, including
language modeling, common-sense reasoning, in-context retrieval, length
extrapolation, and long-context understanding. We further enhance performance
by developing hybrid architectures that combine Gated DeltaNet layers with
sliding window attention or Mamba2 layers, achieving both improved training
efficiency and superior task performance.
Gated DeltaNet, combining gating and delta update mechanisms, outperforms existing models in various tasks and achieves better performance through hybrid architectures with additional attention layers.
Linear Transformers standard Transformers gating adaptive memory control delta update rule parallel training algorithm Gated DeltaNet Mamba2 DeltaNet language modeling common-sense reasoning in-context retrieval length extrapolation long-context understanding sliding window attention