title: MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway
    Dynamic Dense Connections
summary: We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective
  method to address the limitations of residual connections and enhance
  cross-layer information flow in Transformers. Unlike existing dense connection
  approaches with static and shared connection weights, MUDD generates connection
  weights dynamically depending on hidden states at each sequence position and
  for each decoupled input stream (the query, key, value or residual) of a
  Transformer block. MUDD connections can be seamlessly integrated into any
  Transformer architecture to create MUDDFormer. Extensive experiments show that
  MUDDFormer significantly outperforms Transformers across various model
  architectures and scales in language modeling, achieving the performance of
  Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches
  Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B
  in five-shot settings, while adding only 0.23% parameters and 0.4% computation.
  Code in JAX and PyTorch and pre-trained models are available at
  https://github.com/Caiyun-AI/MUDDFormer .
paper_authors._id: 67b5434f2b2ec6908fffe75e
paper_authors.name: Da Xiao
paper_authors._id: 67b5434f2b2ec6908fffe75f
paper_authors.name: Qingye Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:23:45.040Z
paper_authors._id: 634f9c94cdc89e42cc7b194a
paper_authors.avatarUrl: /avatars/42376474afe9b68dc44184c71e210529.svg
paper_authors.fullname: Qingye Meng
paper_authors.type: user
paper_authors.user: Hilbertmeng
paper_authors._id: 67b5434f2b2ec6908fffe760
paper_authors.name: Shengping Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:23:38.331Z
paper_authors._id: 6706a37cca9b1a88fc3951ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Y4GvpKiVgqm_2Kod52_B8.png
paper_authors.fullname: lishengping
paper_authors.type: user
paper_authors.user: lishengping
paper_authors._id: 67b5434f2b2ec6908fffe761
paper_authors.name: Xingyuan Yuan
paper_title: MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway
    Dynamic Dense Connections
paper_summary: We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective
  method to address the limitations of residual connections and enhance
  cross-layer information flow in Transformers. Unlike existing dense connection
  approaches with static and shared connection weights, MUDD generates connection
  weights dynamically depending on hidden states at each sequence position and
  for each decoupled input stream (the query, key, value or residual) of a
  Transformer block. MUDD connections can be seamlessly integrated into any
  Transformer architecture to create MUDDFormer. Extensive experiments show that
  MUDDFormer significantly outperforms Transformers across various model
  architectures and scales in language modeling, achieving the performance of
  Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches
  Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B
  in five-shot settings, while adding only 0.23% parameters and 0.4% computation.
  Code in JAX and PyTorch and pre-trained models are available at
  https://github.com/Caiyun-AI/MUDDFormer .
paper_ai_summary: MUDD connections dynamically generate weights for cross-layer information flow in Transformers, enhancing performance with minimal computational overhead.
paper_ai_keywords: MUltiway Dynamic Dense MUDD residual connections dense connection hidden states sequence position decoupled input stream Transformer block MUDDFormer language modeling pretraining ppl downstream tasks five-shot settings JAX PyTorch pre-trained models