DeepSeekMoE: Towards Ultimate Expert Specialization in
  Mixture-of-Experts Language Models
In the era of large language models, Mixture-of-Experts (MoE) is a promising
architecture for managing computational costs when scaling up model parameters.
However, conventional MoE architectures like GShard, which activate the top-K
out of N experts, face challenges in ensuring expert specialization, i.e.
each expert acquires non-overlapping and focused knowledge. In response, we
propose the DeepSeekMoE architecture towards ultimate expert specialization. It
involves two principal strategies: (1) finely segmenting the experts into mN
ones and activating mK from them, allowing for a more flexible combination of
activated experts; (2) isolating K_s experts as shared ones, aiming at
capturing common knowledge and mitigating redundancy in routed experts.
Starting from a modest scale with 2B parameters, we demonstrate that
DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5
times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly
approaches the performance of its dense counterpart with the same number of
total parameters, which set the upper bound of MoE models. Subsequently, we
scale up DeepSeekMoE to 16B parameters and show that it achieves comparable
performance with LLaMA2 7B, with only about 40% of computations. Further, our
preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently
validate its substantial advantages over the GShard architecture, and show its
performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%)
of computations.
65a0a694db5d37ad5e8bca82
Damai Dai
admin_assigned
2024-01-12T10:06:25.776Z
659389f8de82e1ef7b9a8b13
/avatars/896ed9f4cdbd317493b303d070b7e12a.svg
Damai Dai
user
DeepSeekDDM
65a0a694db5d37ad5e8bca83
Chengqi Deng
65a0a694db5d37ad5e8bca84
Chenggang Zhao
admin_assigned
2024-01-12T10:06:49.782Z
64352c49901c5734bcb72cfa
https://cdn-avatars.huggingface.co/v1/production/uploads/64352c49901c5734bcb72cfa/jGCMbi4JETWoTV47lSp_u.png
Zhao Chenggang
user
zcgamazing
65a0a694db5d37ad5e8bca85
R. X. Xu
65a0a694db5d37ad5e8bca86
Huazuo Gao
admin_assigned
2024-01-12T10:07:01.241Z
64e370be59aa5366642ac329
/avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
Huazuo Gao
user
gaohuazuo
65a0a694db5d37ad5e8bca87
Deli Chen
admin_assigned
2024-01-12T10:07:08.725Z
64ec0b6307987950ae738083
/avatars/e131cd22b55e4b62cc48779e512c3da1.svg
deli chen
user
deli96
65a0a694db5d37ad5e8bca88
Jiashi Li
admin_assigned
2024-01-12T10:07:15.235Z
64fca5f28d50404bc42ca78a
/avatars/ae01ac0296d6ce1277dacb6894f570b8.svg
Jiashi Li
user
Beginlner
65a0a694db5d37ad5e8bca89
Wangding Zeng
admin_assigned
2024-01-12T10:07:27.958Z
659384dec0b1372b2e3c184d
/avatars/fdbff8012fe80d5f2e861cbfa2675125.svg
wangding zeng
user
zwd973-deepseek
65a0a694db5d37ad5e8bca8a
Xingkai Yu
admin_assigned
2024-01-12T10:07:33.898Z
6475b14a37ed88a749e5e48c
/avatars/973f4662023f0bbbc94c01dc3bb3edd3.svg
Xingkai Yu
user
GeeeekExplorer
65a0a694db5d37ad5e8bca8b
Y. Wu
65a0a694db5d37ad5e8bca8c
Zhenda Xie
admin_assigned
2024-01-12T10:07:42.020Z
6539f6ea26df26ecd1393c37
https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
Zhenda Xie
user
zdaxie
65a0a694db5d37ad5e8bca8d
Y. K. Li
65a0a694db5d37ad5e8bca8e
Panpan Huang
65a0a694db5d37ad5e8bca8f
Fuli Luo
admin_assigned
2024-01-12T10:08:04.084Z
6538815d1bdb3c40db94fbfa
https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/id7aSY8JUgKK2agKWLERt.jpeg
Fuli Luo
user
luofuli
65a0a694db5d37ad5e8bca90
Chong Ruan
claimed_verified
2024-08-17T05:38:36.737Z
6398203609f12714ed1935c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
Chong Ruan
user
Chester111
65a0a694db5d37ad5e8bca91
Zhifang Sui
65a0a694db5d37ad5e8bca92
Wenfeng Liang
DeepSeekMoE: Towards Ultimate Expert Specialization in
  Mixture-of-Experts Language Models
In the era of large language models, Mixture-of-Experts (MoE) is a promising
architecture for managing computational costs when scaling up model parameters.
However, conventional MoE architectures like GShard, which activate the top-K
out of N experts, face challenges in ensuring expert specialization, i.e.
each expert acquires non-overlapping and focused knowledge. In response, we
propose the DeepSeekMoE architecture towards ultimate expert specialization. It
involves two principal strategies: (1) finely segmenting the experts into mN
ones and activating mK from them, allowing for a more flexible combination of
activated experts; (2) isolating K_s experts as shared ones, aiming at
capturing common knowledge and mitigating redundancy in routed experts.
Starting from a modest scale with 2B parameters, we demonstrate that
DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5
times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly
approaches the performance of its dense counterpart with the same number of
total parameters, which set the upper bound of MoE models. Subsequently, we
scale up DeepSeekMoE to 16B parameters and show that it achieves comparable
performance with LLaMA2 7B, with only about 40% of computations. Further, our
preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently
validate its substantial advantages over the GShard architecture, and show its
performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%)
of computations.
The DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.
Mixture-of-Experts MoE GShard expert specialization deepSeekMoE shared experts expert parameters computational costs performance dense counterpart LLaMA2 DeepSeek