MH-MoE:Multi-Head Mixture-of-Experts
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by
using the multi-head mechanism to collectively attend to information from
various representation spaces within different experts. In this paper, we
present a novel implementation of MH-MoE that maintains both FLOPs and
parameter parity with sparse Mixture of Experts models. Experimental results on
language models show that the new implementation yields quality improvements
over both vanilla MoE and fine-grained MoE models. Additionally, our
experiments demonstrate that MH-MoE is compatible with 1-bit Large Language
Models (LLMs) such as BitNet.
674546c8c868984c8a473c03
Shaohan Huang
admin_assigned
2024-11-26T22:37:31.415Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
674546c8c868984c8a473c04
Xun Wu
admin_assigned
2024-11-26T22:37:45.155Z
62d1227384bfbee86b6eec56
/avatars/84435f9768a76c0fe9d404dfc2d70be3.svg
Xun Wu
user
YUSHUIWX
674546c8c868984c8a473c05
Shuming Ma
admin_assigned
2024-11-26T22:37:50.443Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
674546c8c868984c8a473c06
Furu Wei
admin_assigned
2024-11-26T22:37:55.841Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
MH-MoE:Multi-Head Mixture-of-Experts
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by
using the multi-head mechanism to collectively attend to information from
various representation spaces within different experts. In this paper, we
present a novel implementation of MH-MoE that maintains both FLOPs and
parameter parity with sparse Mixture of Experts models. Experimental results on
language models show that the new implementation yields quality improvements
over both vanilla MoE and fine-grained MoE models. Additionally, our
experiments demonstrate that MH-MoE is compatible with 1-bit Large Language
Models (LLMs) such as BitNet.
A novel implementation of Multi-Head Mixture-of-Experts maintains efficiency and surpasses traditional models in language performance, even with 1-bit Large Language Models.
Multi-Head Mixture-of-Experts MH-MoE representation spaces experts FLOPs parameter parity sparse Mixture of Experts vanilla MoE fine-grained MoE bitwise Large Language Models BitNet