title: OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
summary: To help the open-source community have a better understanding of
  Mixture-of-Experts (MoE) based large language models (LLMs), we train and
  release OpenMoE, a series of fully open-sourced and reproducible decoder-only
  MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T
  tokens. Our investigation confirms that MoE-based LLMs can offer a more
  favorable cost-effectiveness trade-off than dense LLMs, highlighting the
  potential effectiveness for future LLM development.
    One more important contribution of this study is an in-depth analysis of the
  routing mechanisms within our OpenMoE models, leading to three significant
  findings: Context-Independent Specialization, Early Routing Learning, and
  Drop-towards-the-End. We discovered that routing decisions in MoE models are
  predominantly based on token IDs, with minimal context relevance. The
  token-to-expert assignments are determined early in the pre-training phase and
  remain largely unchanged. This imperfect routing can result in performance
  degradation, particularly in sequential tasks like multi-turn conversations,
  where tokens appearing later in a sequence are more likely to be dropped.
  Finally, we rethink our design based on the above-mentioned observations and
  analysis. To facilitate future MoE LLM development, we propose potential
  strategies for mitigating the issues we found and further improving
  off-the-shelf MoE LLM designs.
paper_authors._id: 65c1b2babd8a99477a35858b
paper_authors.name: Fuzhao Xue
paper_authors._id: 65c1b2babd8a99477a35858c
paper_authors.name: Zian Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:54:53.178Z
paper_authors._id: 6371e32aafbe42caa5ad9189
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371e32aafbe42caa5ad9189/Frqjr_L1BGyg4yWwVXKiv.jpeg
paper_authors.fullname: Zheng Zian(Andy)
paper_authors.type: user
paper_authors.user: OrionZheng
paper_authors._id: 65c1b2babd8a99477a35858d
paper_authors.name: Yao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T07:42:27.417Z
paper_authors._id: 636c929d5ef5a1a07db4e785
paper_authors.avatarUrl: /avatars/1bb48dbba964f7fa84036586032684cf.svg
paper_authors.fullname: Yao Fu
paper_authors.type: user
paper_authors.user: yaofu
paper_authors._id: 65c1b2babd8a99477a35858e
paper_authors.name: Jinjie Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:30:48.058Z
paper_authors._id: 62a7362fd1e7a011fd4e31a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
paper_authors.fullname: Jinjie Ni
paper_authors.type: user
paper_authors.user: jinjieni
paper_authors._id: 65c1b2babd8a99477a35858f
paper_authors.name: Zangwei Zheng
paper_authors._id: 65c1b2babd8a99477a358590
paper_authors.name: Wangchunshu Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:19:36.248Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 65c1b2babd8a99477a358591
paper_authors.name: Yang You
paper_title: OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
paper_summary: To help the open-source community have a better understanding of
  Mixture-of-Experts (MoE) based large language models (LLMs), we train and
  release OpenMoE, a series of fully open-sourced and reproducible decoder-only
  MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T
  tokens. Our investigation confirms that MoE-based LLMs can offer a more
  favorable cost-effectiveness trade-off than dense LLMs, highlighting the
  potential effectiveness for future LLM development.
    One more important contribution of this study is an in-depth analysis of the
  routing mechanisms within our OpenMoE models, leading to three significant
  findings: Context-Independent Specialization, Early Routing Learning, and
  Drop-towards-the-End. We discovered that routing decisions in MoE models are
  predominantly based on token IDs, with minimal context relevance. The
  token-to-expert assignments are determined early in the pre-training phase and
  remain largely unchanged. This imperfect routing can result in performance
  degradation, particularly in sequential tasks like multi-turn conversations,
  where tokens appearing later in a sequence are more likely to be dropped.
  Finally, we rethink our design based on the above-mentioned observations and
  analysis. To facilitate future MoE LLM development, we propose potential
  strategies for mitigating the issues we found and further improving
  off-the-shelf MoE LLM designs.
paper_ai_keywords: Mixture-of-Experts (MoE) large language models (LLMs) decoder-only MoE LLMs routing mechanisms Context-Independent Specialization Early Routing Learning Drop-towards-the-End token IDs token-to-expert assignments multi-turn conversations off-the-shelf MoE LLM designs