title: MH-MoE:Multi-Head Mixture-of-Experts
summary: Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by
  using the multi-head mechanism to collectively attend to information from
  various representation spaces within different experts. In this paper, we
  present a novel implementation of MH-MoE that maintains both FLOPs and
  parameter parity with sparse Mixture of Experts models. Experimental results on
  language models show that the new implementation yields quality improvements
  over both vanilla MoE and fine-grained MoE models. Additionally, our
  experiments demonstrate that MH-MoE is compatible with 1-bit Large Language
  Models (LLMs) such as BitNet.
paper_authors._id: 674546c8c868984c8a473c03
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:37:31.415Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 674546c8c868984c8a473c04
paper_authors.name: Xun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:37:45.155Z
paper_authors._id: 62d1227384bfbee86b6eec56
paper_authors.avatarUrl: /avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
paper_authors.fullname: Xun Wu
paper_authors.type: user
paper_authors.user: YUSHUIWX
paper_authors._id: 674546c8c868984c8a473c05
paper_authors.name: Shuming Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:37:50.443Z
paper_authors._id: 613f07f40153aafa379775f2
paper_authors.avatarUrl: /avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
paper_authors.fullname: Shuming Ma
paper_authors.type: user
paper_authors.user: shumingma
paper_authors._id: 674546c8c868984c8a473c06
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:37:55.841Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: MH-MoE:Multi-Head Mixture-of-Experts
paper_summary: Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by
  using the multi-head mechanism to collectively attend to information from
  various representation spaces within different experts. In this paper, we
  present a novel implementation of MH-MoE that maintains both FLOPs and
  parameter parity with sparse Mixture of Experts models. Experimental results on
  language models show that the new implementation yields quality improvements
  over both vanilla MoE and fine-grained MoE models. Additionally, our
  experiments demonstrate that MH-MoE is compatible with 1-bit Large Language
  Models (LLMs) such as BitNet.
paper_ai_summary: A novel implementation of Multi-Head Mixture-of-Experts maintains efficiency and surpasses traditional models in language performance, even with 1-bit Large Language Models.
paper_ai_keywords: Multi-Head Mixture-of-Experts MH-MoE representation spaces experts FLOPs parameter parity sparse Mixture of Experts vanilla MoE fine-grained MoE bitwise Large Language Models BitNet