Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
  Language Models
In this technical report, we introduce the training methodologies implemented
in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
large language model (LLM) with 146 billion parameters and 16 experts. It is
initialized from the pre-existing dense checkpoints of our Skywork-13B model.
We explore the comparative effectiveness of upcycling versus training from
scratch initializations. Our findings suggest that the choice between these two
approaches should consider both the performance of the existing dense
checkpoints and the MoE training budget. We highlight two innovative
techniques: gating logit normalization, which improves expert diversification,
and adaptive auxiliary loss coefficients, allowing for layer-specific
adjustment of auxiliary loss coefficients. Our experimental results validate
the effectiveness of these methods. Leveraging these techniques and insights,
we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
corpus. The evaluation results demonstrate that our model delivers strong
performance across a wide range of benchmarks.
666919d303e3c114f06b8f15
Tianwen Wei
admin_assigned
2024-06-12T09:14:32.771Z
63d8773a635499f21644601b
/avatars/07efea204d172e2844dbacc418131da2.svg
Wei Tianwen
user
weitianwen
666919d303e3c114f06b8f16
Bo Zhu
666919d303e3c114f06b8f17
Liang Zhao
claimed_verified
2024-06-12T09:21:14.969Z
64996ce2ed09df36eaf236ea
https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
zhaoliang
user
zhao1iang
666919d303e3c114f06b8f18
Cheng Cheng
claimed_verified
2024-06-12T09:21:16.941Z
64535b71bcbd25618f7655da
https://cdn-avatars.huggingface.co/v1/production/uploads/64535b71bcbd25618f7655da/vDc8nrYQd4rpYgXLWOuIx.jpeg
cheng cheng
user
chengtbf
666919d303e3c114f06b8f19
Biye Li
666919d303e3c114f06b8f1a
Weiwei Lü
666919d303e3c114f06b8f1b
Peng Cheng
claimed_verified
2024-06-12T09:24:00.021Z
636b8f4429668f04eaf5d015
/avatars/84b6209914cb83af7cfce2ffb337d7bd.svg
CPFLAME
user
CPFLAME
666919d303e3c114f06b8f1c
Jianhao Zhang
admin_assigned
2024-06-12T09:24:18.827Z
64235adf23b98d5ee6e9ca85
https://cdn-avatars.huggingface.co/v1/production/uploads/64235adf23b98d5ee6e9ca85/m_7uwLaAkTXyt3qB2pPLA.png
Jianhao Zhang
user
Heerozh
666919d303e3c114f06b8f1d
Xiaoyu Zhang
claimed_verified
2024-06-12T11:50:52.655Z
63647823a1503c61efd4fbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63647823a1503c61efd4fbcb/Df5U59Imobmr9S8wDf77I.png
Yi Wang
user
BBuf
666919d303e3c114f06b8f1e
Liang Zeng
claimed_verified
2024-06-12T09:31:25.323Z
6621efe1a6eec3ad03e38759
/avatars/c35acce69f244ec0833dffd53eedf6a3.svg
Liang Zeng
user
zengliangcs
666919d303e3c114f06b8f1f
Xiaokun Wang
claimed_verified
2025-01-09T10:12:40.976Z
62be9b5aae56e75e4d689e7c
/avatars/6772bc09d6eeb4e86b1210481be91720.svg
wangxiaokun
user
shawn0wang
666919d303e3c114f06b8f20
Yutuan Ma
666919d303e3c114f06b8f21
Rui Hu
666919d303e3c114f06b8f22
Shuicheng Yan
666919d303e3c114f06b8f23
Han Fang
666919d303e3c114f06b8f24
Yahui Zhou
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
  Language Models
In this technical report, we introduce the training methodologies implemented
in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
large language model (LLM) with 146 billion parameters and 16 experts. It is
initialized from the pre-existing dense checkpoints of our Skywork-13B model.
We explore the comparative effectiveness of upcycling versus training from
scratch initializations. Our findings suggest that the choice between these two
approaches should consider both the performance of the existing dense
checkpoints and the MoE training budget. We highlight two innovative
techniques: gating logit normalization, which improves expert diversification,
and adaptive auxiliary loss coefficients, allowing for layer-specific
adjustment of auxiliary loss coefficients. Our experimental results validate
the effectiveness of these methods. Leveraging these techniques and insights,
we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
corpus. The evaluation results demonstrate that our model delivers strong
performance across a wide range of benchmarks.
Skywork-MoE, a high-performance mixture-of-experts language model, uses upcycling and innovative techniques like gating logit normalization and adaptive auxiliary loss coefficients to achieve strong performance across benchmarks.
mixture-of-experts large language model MoE gating logit normalization adaptive auxiliary loss coefficients