title: Union of Experts: Adapting Hierarchical Routing to Equivalently
    Decomposed Transformer
summary: Mixture-of-Experts (MoE) enhances model performance while maintaining
  computational efficiency, making it well-suited for large-scale applications.
  However, expert in exist MoE paradigm works as an individual, thereby lacking
  high-quality expert interactions. Moreover, they have not been effectively
  extended to attention block, which constrains further efficiency improvements.
  To tackle these issues, we propose Union-of-Experts (UoE), which decomposes
  transformer into an equitant group of experts, and then implement dynamic
  routing on input data and experts. Our approach advances MoE design with three
  key innovations: (1) We conducted equitant expert decomposition on both MLP
  blocks and attention blocks based on matrix partition in tensor parallelism.
  (2) We developed two routing paradigms: patch wise data selection and expert
  selection, to apply routing across different levels. (3) We design the
  architecture of UoE model, including Selective Multi-Head Attention (SMHA) and
  Union-of-MLP-Experts (UoME). (4) We develop parallel implementation of UoE's
  routing and computation operation, and optimize efficiency based on the
  hardware processing analysis. The experiments demonstrate that the model
  employed with UoE surpass Full Attention, state-of-art MoEs and efficient
  transformers in several tasks across image and natural language domains. The
  source codes are available at https://github.com/YujiaoYang-work/UoE.
paper_authors._id: 67c9e3b7668d5bddb4d36a46
paper_authors.name: Yujiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:10:00.268Z
paper_authors._id: 67a480dadba32bb665f5aeae
paper_authors.avatarUrl: /avatars/ee8d9481a9e86f5e9803bd934a095d61.svg
paper_authors.fullname: Yujiao Yang
paper_authors.type: user
paper_authors.user: yjyangwork
paper_authors._id: 67c9e3b7668d5bddb4d36a47
paper_authors.name: Jing Lian
paper_authors._id: 67c9e3b7668d5bddb4d36a48
paper_authors.name: Linhui Li
paper_title: Union of Experts: Adapting Hierarchical Routing to Equivalently
    Decomposed Transformer
paper_summary: Mixture-of-Experts (MoE) enhances model performance while maintaining
  computational efficiency, making it well-suited for large-scale applications.
  However, expert in exist MoE paradigm works as an individual, thereby lacking
  high-quality expert interactions. Moreover, they have not been effectively
  extended to attention block, which constrains further efficiency improvements.
  To tackle these issues, we propose Union-of-Experts (UoE), which decomposes
  transformer into an equitant group of experts, and then implement dynamic
  routing on input data and experts. Our approach advances MoE design with three
  key innovations: (1) We conducted equitant expert decomposition on both MLP
  blocks and attention blocks based on matrix partition in tensor parallelism.
  (2) We developed two routing paradigms: patch wise data selection and expert
  selection, to apply routing across different levels. (3) We design the
  architecture of UoE model, including Selective Multi-Head Attention (SMHA) and
  Union-of-MLP-Experts (UoME). (4) We develop parallel implementation of UoE's
  routing and computation operation, and optimize efficiency based on the
  hardware processing analysis. The experiments demonstrate that the model
  employed with UoE surpass Full Attention, state-of-art MoEs and efficient
  transformers in several tasks across image and natural language domains. The
  source codes are available at https://github.com/YujiaoYang-work/UoE.
paper_ai_keywords: Union-of-Experts (UoE) Mixture-of-Experts (MoE) transformer equitant expert decomposition matrix partition tensor parallelism dynamic routing MLP blocks attention blocks routing paradigms patch wise data selection expert selection Selective Multi-Head Attention (SMHA) Union-of-MLP-Experts (UoME) Full Attention state-of-art MoEs efficient transformers hardware processing analysis