title: Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
    Language Models
summary: In this technical report, we introduce the training methodologies implemented
  in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
  large language model (LLM) with 146 billion parameters and 16 experts. It is
  initialized from the pre-existing dense checkpoints of our Skywork-13B model.
  We explore the comparative effectiveness of upcycling versus training from
  scratch initializations. Our findings suggest that the choice between these two
  approaches should consider both the performance of the existing dense
  checkpoints and the MoE training budget. We highlight two innovative
  techniques: gating logit normalization, which improves expert diversification,
  and adaptive auxiliary loss coefficients, allowing for layer-specific
  adjustment of auxiliary loss coefficients. Our experimental results validate
  the effectiveness of these methods. Leveraging these techniques and insights,
  we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
  corpus. The evaluation results demonstrate that our model delivers strong
  performance across a wide range of benchmarks.
paper_authors._id: 666919d303e3c114f06b8f15
paper_authors.name: Tianwen Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:14:32.771Z
paper_authors._id: 63d8773a635499f21644601b
paper_authors.avatarUrl: /avatars/07efea204d172e2844dbacc418131da2.svg
paper_authors.fullname: Wei Tianwen
paper_authors.type: user
paper_authors.user: weitianwen
paper_authors._id: 666919d303e3c114f06b8f16
paper_authors.name: Bo Zhu
paper_authors._id: 666919d303e3c114f06b8f17
paper_authors.name: Liang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:21:14.969Z
paper_authors._id: 64996ce2ed09df36eaf236ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
paper_authors.fullname: zhaoliang
paper_authors.type: user
paper_authors.user: zhao1iang
paper_authors._id: 666919d303e3c114f06b8f18
paper_authors.name: Cheng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:21:16.941Z
paper_authors._id: 64535b71bcbd25618f7655da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64535b71bcbd25618f7655da/vDc8nrYQd4rpYgXLWOuIx.jpeg
paper_authors.fullname: cheng cheng
paper_authors.type: user
paper_authors.user: chengtbf
paper_authors._id: 666919d303e3c114f06b8f19
paper_authors.name: Biye Li
paper_authors._id: 666919d303e3c114f06b8f1a
paper_authors.name: Weiwei Lü
paper_authors._id: 666919d303e3c114f06b8f1b
paper_authors.name: Peng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:24:00.021Z
paper_authors._id: 636b8f4429668f04eaf5d015
paper_authors.avatarUrl: /avatars/84b6209914cb83af7cfce2ffb337d7bd.svg
paper_authors.fullname: CPFLAME
paper_authors.type: user
paper_authors.user: CPFLAME
paper_authors._id: 666919d303e3c114f06b8f1c
paper_authors.name: Jianhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T09:24:18.827Z
paper_authors._id: 64235adf23b98d5ee6e9ca85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64235adf23b98d5ee6e9ca85/m_7uwLaAkTXyt3qB2pPLA.png
paper_authors.fullname: Jianhao Zhang
paper_authors.type: user
paper_authors.user: Heerozh
paper_authors._id: 666919d303e3c114f06b8f1d
paper_authors.name: Xiaoyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T11:50:52.655Z
paper_authors._id: 63647823a1503c61efd4fbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63647823a1503c61efd4fbcb/fOn74vqVyvR4nCP_5sxQK.png
paper_authors.fullname: Yi Wang
paper_authors.type: user
paper_authors.user: BBuf
paper_authors._id: 666919d303e3c114f06b8f1e
paper_authors.name: Liang Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:31:25.323Z
paper_authors._id: 6621efe1a6eec3ad03e38759
paper_authors.avatarUrl: /avatars/c35acce69f244ec0833dffd53eedf6a3.svg
paper_authors.fullname: Liang Zeng
paper_authors.type: user
paper_authors.user: zengliangcs
paper_authors._id: 666919d303e3c114f06b8f1f
paper_authors.name: Xiaokun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:12:40.976Z
paper_authors._id: 62be9b5aae56e75e4d689e7c
paper_authors.avatarUrl: /avatars/6772bc09d6eeb4e86b1210481be91720.svg
paper_authors.fullname: wangxiaokun
paper_authors.type: user
paper_authors.user: shawn0wang
paper_authors._id: 666919d303e3c114f06b8f20
paper_authors.name: Yutuan Ma
paper_authors._id: 666919d303e3c114f06b8f21
paper_authors.name: Rui Hu
paper_authors._id: 666919d303e3c114f06b8f22
paper_authors.name: Shuicheng Yan
paper_authors._id: 666919d303e3c114f06b8f23
paper_authors.name: Han Fang
paper_authors._id: 666919d303e3c114f06b8f24
paper_authors.name: Yahui Zhou
paper_title: Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
    Language Models
paper_summary: In this technical report, we introduce the training methodologies implemented
  in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
  large language model (LLM) with 146 billion parameters and 16 experts. It is
  initialized from the pre-existing dense checkpoints of our Skywork-13B model.
  We explore the comparative effectiveness of upcycling versus training from
  scratch initializations. Our findings suggest that the choice between these two
  approaches should consider both the performance of the existing dense
  checkpoints and the MoE training budget. We highlight two innovative
  techniques: gating logit normalization, which improves expert diversification,
  and adaptive auxiliary loss coefficients, allowing for layer-specific
  adjustment of auxiliary loss coefficients. Our experimental results validate
  the effectiveness of these methods. Leveraging these techniques and insights,
  we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
  corpus. The evaluation results demonstrate that our model delivers strong
  performance across a wide range of benchmarks.
paper_ai_keywords: mixture-of-experts (MoE) large language model (LLM) pre-existing dense checkpoints upcycling training from scratch initializations gating logit normalization adaptive auxiliary loss coefficients auxiliary loss coefficients layer-specific adjustment SkyPile corpus