title: Multi-Head Mixture-of-Experts
summary: Sparse Mixtures of Experts (SMoE) scales model capacity without significant
  increases in training and inference costs, but exhibits the following two
  issues: (1) Low expert activation, where only a small subset of experts are
  activated for optimization. (2) Lacking fine-grained analytical capabilities
  for multiple semantic concepts within individual tokens. We propose Multi-Head
  Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
  token into multiple sub-tokens. These sub-tokens are then assigned to and
  processed by a diverse set of experts in parallel, and seamlessly reintegrated
  into the original token form. The multi-head mechanism enables the model to
  collectively attend to information from various representation spaces within
  different experts, while significantly enhances expert activation, thus deepens
  context understanding and alleviate overfitting. Moreover, our MH-MoE is
  straightforward to implement and decouples from other SMoE optimization
  methods, making it easy to integrate with other SMoE models for enhanced
  performance. Extensive experimental results across three tasks: English-focused
  language modeling, Multi-lingual language modeling and Masked multi-modality
  modeling tasks, demonstrate the effectiveness of MH-MoE.
paper_authors._id: 662866ae623aebe1250bd0c0
paper_authors.name: Xun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:11:13.519Z
paper_authors._id: 62d1227384bfbee86b6eec56
paper_authors.avatarUrl: /avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
paper_authors.fullname: Xun Wu
paper_authors.type: user
paper_authors.user: YUSHUIWX
paper_authors._id: 662866ae623aebe1250bd0c1
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:27:02.896Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 662866ae623aebe1250bd0c2
paper_authors.name: Wenhui Wang
paper_authors._id: 662866ae623aebe1250bd0c3
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:27:35.961Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Multi-Head Mixture-of-Experts
paper_summary: Sparse Mixtures of Experts (SMoE) scales model capacity without significant
  increases in training and inference costs, but exhibits the following two
  issues: (1) Low expert activation, where only a small subset of experts are
  activated for optimization. (2) Lacking fine-grained analytical capabilities
  for multiple semantic concepts within individual tokens. We propose Multi-Head
  Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each
  token into multiple sub-tokens. These sub-tokens are then assigned to and
  processed by a diverse set of experts in parallel, and seamlessly reintegrated
  into the original token form. The multi-head mechanism enables the model to
  collectively attend to information from various representation spaces within
  different experts, while significantly enhances expert activation, thus deepens
  context understanding and alleviate overfitting. Moreover, our MH-MoE is
  straightforward to implement and decouples from other SMoE optimization
  methods, making it easy to integrate with other SMoE models for enhanced
  performance. Extensive experimental results across three tasks: English-focused
  language modeling, Multi-lingual language modeling and Masked multi-modality
  modeling tasks, demonstrate the effectiveness of MH-MoE.
paper_ai_summary: MH-MoE enhances SMoE by splitting tokens into sub-tokens processed by diverse experts in parallel, improving expert activation, context understanding, and overfitting, and demonstrating effectiveness across various modeling tasks.
paper_ai_keywords: Sparse Mixtures of Experts (SMoE) Multi-Head Mixture-of-Experts (MH-MoE) expert activation semantic concepts sub-tokens context understanding overfitting English-focused language modeling Multi-lingual language modeling Masked multi-modality modeling