UMoE: Unifying Attention and FFN with Shared Experts
Sparse Mixture of Experts (MoE) architectures have emerged as a promising
approach for scaling Transformer models. While initial works primarily
incorporated MoE into feed-forward network (FFN) layers, recent studies have
explored extending the MoE paradigm to attention layers to enhance model
performance. However, existing attention-based MoE layers require specialized
implementations and demonstrate suboptimal performance compared to their
FFN-based counterparts. In this paper, we aim to unify the MoE designs in
attention and FFN layers by introducing a novel reformulation of the attention
mechanism, revealing an underlying FFN-like structure within attention modules.
Our proposed architecture, UMoE, achieves superior performance through
attention-based MoE layers while enabling efficient parameter sharing between
FFN and attention components.
682307ad692919b8cc2a7912
Yuanhang Yang
claimed_verified
2025-05-13T08:57:51.750Z
621731ee29500f41901123d5
/avatars/ec413c33eb77b67dba35e03c4857f222.svg
YUANHANG YANG
user
ysngkil
682307ad692919b8cc2a7913
Chaozheng Wang
682307ad692919b8cc2a7914
Jing Li
UMoE: Unifying Attention and FFN with Shared Experts
Sparse Mixture of Experts (MoE) architectures have emerged as a promising
approach for scaling Transformer models. While initial works primarily
incorporated MoE into feed-forward network (FFN) layers, recent studies have
explored extending the MoE paradigm to attention layers to enhance model
performance. However, existing attention-based MoE layers require specialized
implementations and demonstrate suboptimal performance compared to their
FFN-based counterparts. In this paper, we aim to unify the MoE designs in
attention and FFN layers by introducing a novel reformulation of the attention
mechanism, revealing an underlying FFN-like structure within attention modules.
Our proposed architecture, UMoE, achieves superior performance through
attention-based MoE layers while enabling efficient parameter sharing between
FFN and attention components.
A novel Sparse Mixture of Experts (MoE) architecture unifies MoE designs in attention and feed-forward layers, enhancing model performance and enabling efficient parameter sharing.
Sparse Mixture of Experts MoE Transformer models feed-forward network FFN layers attention layers attention mechanism UMoE parameter sharing