title: UMoE: Unifying Attention and FFN with Shared Experts
summary: Sparse Mixture of Experts (MoE) architectures have emerged as a promising
  approach for scaling Transformer models. While initial works primarily
  incorporated MoE into feed-forward network (FFN) layers, recent studies have
  explored extending the MoE paradigm to attention layers to enhance model
  performance. However, existing attention-based MoE layers require specialized
  implementations and demonstrate suboptimal performance compared to their
  FFN-based counterparts. In this paper, we aim to unify the MoE designs in
  attention and FFN layers by introducing a novel reformulation of the attention
  mechanism, revealing an underlying FFN-like structure within attention modules.
  Our proposed architecture, UMoE, achieves superior performance through
  attention-based MoE layers while enabling efficient parameter sharing between
  FFN and attention components.
paper_authors._id: 682307ad692919b8cc2a7912
paper_authors.name: Yuanhang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:57:51.750Z
paper_authors._id: 621731ee29500f41901123d5
paper_authors.avatarUrl: /avatars/ec413c33eb77b67dba35e03c4857f222.svg
paper_authors.fullname: YUANHANG YANG
paper_authors.type: user
paper_authors.user: ysngkil
paper_authors._id: 682307ad692919b8cc2a7913
paper_authors.name: Chaozheng Wang
paper_authors._id: 682307ad692919b8cc2a7914
paper_authors.name: Jing Li
paper_title: UMoE: Unifying Attention and FFN with Shared Experts
paper_summary: Sparse Mixture of Experts (MoE) architectures have emerged as a promising
  approach for scaling Transformer models. While initial works primarily
  incorporated MoE into feed-forward network (FFN) layers, recent studies have
  explored extending the MoE paradigm to attention layers to enhance model
  performance. However, existing attention-based MoE layers require specialized
  implementations and demonstrate suboptimal performance compared to their
  FFN-based counterparts. In this paper, we aim to unify the MoE designs in
  attention and FFN layers by introducing a novel reformulation of the attention
  mechanism, revealing an underlying FFN-like structure within attention modules.
  Our proposed architecture, UMoE, achieves superior performance through
  attention-based MoE layers while enabling efficient parameter sharing between
  FFN and attention components.
paper_ai_summary: A novel Sparse Mixture of Experts (MoE) architecture unifies MoE designs in attention and feed-forward layers, enhancing model performance and enabling efficient parameter sharing.
paper_ai_keywords: Sparse Mixture of Experts MoE Transformer models feed-forward network FFN layers attention layers attention mechanism UMoE parameter sharing