OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
To help the open-source community have a better understanding of
Mixture-of-Experts (MoE) based large language models (LLMs), we train and
release OpenMoE, a series of fully open-sourced and reproducible decoder-only
MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T
tokens. Our investigation confirms that MoE-based LLMs can offer a more
favorable cost-effectiveness trade-off than dense LLMs, highlighting the
potential effectiveness for future LLM development.
  One more important contribution of this study is an in-depth analysis of the
routing mechanisms within our OpenMoE models, leading to three significant
findings: Context-Independent Specialization, Early Routing Learning, and
Drop-towards-the-End. We discovered that routing decisions in MoE models are
predominantly based on token IDs, with minimal context relevance. The
token-to-expert assignments are determined early in the pre-training phase and
remain largely unchanged. This imperfect routing can result in performance
degradation, particularly in sequential tasks like multi-turn conversations,
where tokens appearing later in a sequence are more likely to be dropped.
Finally, we rethink our design based on the above-mentioned observations and
analysis. To facilitate future MoE LLM development, we propose potential
strategies for mitigating the issues we found and further improving
off-the-shelf MoE LLM designs.
65c1b2babd8a99477a35858b
Fuzhao Xue
65c1b2babd8a99477a35858c
Zian Zheng
claimed_verified
2024-10-21T07:54:53.178Z
6371e32aafbe42caa5ad9189
https://cdn-avatars.huggingface.co/v1/production/uploads/6371e32aafbe42caa5ad9189/Frqjr_L1BGyg4yWwVXKiv.jpeg
Zheng Zian(Andy)
user
OrionZheng
65c1b2babd8a99477a35858d
Yao Fu
claimed_verified
2024-04-26T07:42:27.417Z
636c929d5ef5a1a07db4e785
/avatars/1bb48dbba964f7fa84036586032684cf.svg
Yao Fu
user
yaofu
65c1b2babd8a99477a35858e
Jinjie Ni
claimed_verified
2024-06-12T07:30:48.058Z
62a7362fd1e7a011fd4e31a7
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
Jinjie Ni
user
jinjieni
65c1b2babd8a99477a35858f
Zangwei Zheng
65c1b2babd8a99477a358590
Wangchunshu Zhou
claimed_verified
2024-02-07T11:19:36.248Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
65c1b2babd8a99477a358591
Yang You
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
To help the open-source community have a better understanding of
Mixture-of-Experts (MoE) based large language models (LLMs), we train and
release OpenMoE, a series of fully open-sourced and reproducible decoder-only
MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T
tokens. Our investigation confirms that MoE-based LLMs can offer a more
favorable cost-effectiveness trade-off than dense LLMs, highlighting the
potential effectiveness for future LLM development.
  One more important contribution of this study is an in-depth analysis of the
routing mechanisms within our OpenMoE models, leading to three significant
findings: Context-Independent Specialization, Early Routing Learning, and
Drop-towards-the-End. We discovered that routing decisions in MoE models are
predominantly based on token IDs, with minimal context relevance. The
token-to-expert assignments are determined early in the pre-training phase and
remain largely unchanged. This imperfect routing can result in performance
degradation, particularly in sequential tasks like multi-turn conversations,
where tokens appearing later in a sequence are more likely to be dropped.
Finally, we rethink our design based on the above-mentioned observations and
analysis. To facilitate future MoE LLM development, we propose potential
strategies for mitigating the issues we found and further improving
off-the-shelf MoE LLM designs.
OpenMoE, a series of open-sourced MoE-based decoder-only LLMs, demonstrates improved cost-effectiveness over dense LLMs and reveals insights into MoE routing mechanisms, including context-independent specialization, early routing learning, and drop-towards-the-end issues.
Mixture-of-Experts (MoE) decoder-only large language models (LLMs) OpenMoE routing mechanisms Context-Independent Specialization Early Routing Learning Drop-towards-the-End