title: DeepSeekMoE: Towards Ultimate Expert Specialization in
    Mixture-of-Experts Language Models
summary: In the era of large language models, Mixture-of-Experts (MoE) is a promising
  architecture for managing computational costs when scaling up model parameters.
  However, conventional MoE architectures like GShard, which activate the top-K
  out of N experts, face challenges in ensuring expert specialization, i.e.
  each expert acquires non-overlapping and focused knowledge. In response, we
  propose the DeepSeekMoE architecture towards ultimate expert specialization. It
  involves two principal strategies: (1) finely segmenting the experts into mN
  ones and activating mK from them, allowing for a more flexible combination of
  activated experts; (2) isolating K_s experts as shared ones, aiming at
  capturing common knowledge and mitigating redundancy in routed experts.
  Starting from a modest scale with 2B parameters, we demonstrate that
  DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5
  times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly
  approaches the performance of its dense counterpart with the same number of
  total parameters, which set the upper bound of MoE models. Subsequently, we
  scale up DeepSeekMoE to 16B parameters and show that it achieves comparable
  performance with LLaMA2 7B, with only about 40% of computations. Further, our
  preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently
  validate its substantial advantages over the GShard architecture, and show its
  performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%)
  of computations.
paper_authors._id: 65a0a694db5d37ad5e8bca82
paper_authors.name: Damai Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:06:25.776Z
paper_authors._id: 659389f8de82e1ef7b9a8b13
paper_authors.avatarUrl: /avatars/896ed9f4cdbd317493b303d070b7e12a.svg
paper_authors.fullname: Damai Dai
paper_authors.type: user
paper_authors.user: DeepSeekDDM
paper_authors._id: 65a0a694db5d37ad5e8bca83
paper_authors.name: Chengqi Deng
paper_authors._id: 65a0a694db5d37ad5e8bca84
paper_authors.name: Chenggang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:06:49.782Z
paper_authors._id: 64352c49901c5734bcb72cfa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64352c49901c5734bcb72cfa/jGCMbi4JETWoTV47lSp_u.png
paper_authors.fullname: Zhao Chenggang
paper_authors.type: user
paper_authors.user: zcgamazing
paper_authors._id: 65a0a694db5d37ad5e8bca85
paper_authors.name: R. X. Xu
paper_authors._id: 65a0a694db5d37ad5e8bca86
paper_authors.name: Huazuo Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:01.241Z
paper_authors._id: 64e370be59aa5366642ac329
paper_authors.avatarUrl: /avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
paper_authors.fullname: Huazuo Gao
paper_authors.type: user
paper_authors.user: gaohuazuo
paper_authors._id: 65a0a694db5d37ad5e8bca87
paper_authors.name: Deli Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:08.725Z
paper_authors._id: 64ec0b6307987950ae738083
paper_authors.avatarUrl: /avatars/e131cd22b55e4b62cc48779e512c3da1.svg
paper_authors.fullname: deli chen
paper_authors.type: user
paper_authors.user: deli96
paper_authors._id: 65a0a694db5d37ad5e8bca88
paper_authors.name: Jiashi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:15.235Z
paper_authors._id: 64fca5f28d50404bc42ca78a
paper_authors.avatarUrl: /avatars/ae01ac0296d6ce1277dacb6894f570b8.svg
paper_authors.fullname: Jiashi Li
paper_authors.type: user
paper_authors.user: Beginlner
paper_authors._id: 65a0a694db5d37ad5e8bca89
paper_authors.name: Wangding Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:27.958Z
paper_authors._id: 659384dec0b1372b2e3c184d
paper_authors.avatarUrl: /avatars/fdbff8012fe80d5f2e861cbfa2675125.svg
paper_authors.fullname: wangding zeng
paper_authors.type: user
paper_authors.user: zwd973-deepseek
paper_authors._id: 65a0a694db5d37ad5e8bca8a
paper_authors.name: Xingkai Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:33.898Z
paper_authors._id: 6475b14a37ed88a749e5e48c
paper_authors.avatarUrl: /avatars/973f4662023f0bbbc94c01dc3bb3edd3.svg
paper_authors.fullname: Xingkai Yu
paper_authors.type: user
paper_authors.user: GeeeekExplorer
paper_authors._id: 65a0a694db5d37ad5e8bca8b
paper_authors.name: Y. Wu
paper_authors._id: 65a0a694db5d37ad5e8bca8c
paper_authors.name: Zhenda Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:07:42.020Z
paper_authors._id: 6539f6ea26df26ecd1393c37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
paper_authors.fullname: Zhenda Xie
paper_authors.type: user
paper_authors.user: zdaxie
paper_authors._id: 65a0a694db5d37ad5e8bca8d
paper_authors.name: Y. K. Li
paper_authors._id: 65a0a694db5d37ad5e8bca8e
paper_authors.name: Panpan Huang
paper_authors._id: 65a0a694db5d37ad5e8bca8f
paper_authors.name: Fuli Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:08:04.084Z
paper_authors._id: 6538815d1bdb3c40db94fbfa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/id7aSY8JUgKK2agKWLERt.jpeg
paper_authors.fullname: Fuli Luo
paper_authors.type: user
paper_authors.user: luofuli
paper_authors._id: 65a0a694db5d37ad5e8bca90
paper_authors.name: Chong Ruan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-17T05:38:36.737Z
paper_authors._id: 6398203609f12714ed1935c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
paper_authors.fullname: Chong Ruan
paper_authors.type: user
paper_authors.user: Chester111
paper_authors._id: 65a0a694db5d37ad5e8bca91
paper_authors.name: Zhifang Sui
paper_authors._id: 65a0a694db5d37ad5e8bca92
paper_authors.name: Wenfeng Liang
paper_title: DeepSeekMoE: Towards Ultimate Expert Specialization in
    Mixture-of-Experts Language Models
paper_summary: In the era of large language models, Mixture-of-Experts (MoE) is a promising
  architecture for managing computational costs when scaling up model parameters.
  However, conventional MoE architectures like GShard, which activate the top-K
  out of N experts, face challenges in ensuring expert specialization, i.e.
  each expert acquires non-overlapping and focused knowledge. In response, we
  propose the DeepSeekMoE architecture towards ultimate expert specialization. It
  involves two principal strategies: (1) finely segmenting the experts into mN
  ones and activating mK from them, allowing for a more flexible combination of
  activated experts; (2) isolating K_s experts as shared ones, aiming at
  capturing common knowledge and mitigating redundancy in routed experts.
  Starting from a modest scale with 2B parameters, we demonstrate that
  DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5
  times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly
  approaches the performance of its dense counterpart with the same number of
  total parameters, which set the upper bound of MoE models. Subsequently, we
  scale up DeepSeekMoE to 16B parameters and show that it achieves comparable
  performance with LLaMA2 7B, with only about 40% of computations. Further, our
  preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently
  validate its substantial advantages over the GShard architecture, and show its
  performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%)
  of computations.
paper_ai_keywords: Mixture-of-Experts (MoE) GShard DeepSeekMoE expert specialization finely segmenting activated experts shared experts common knowledge redundancy dense counterpart LLaMA2 computational requirements parameter-efficient performance bounds