title: GRIN: GRadient-INformed MoE
summary: Mixture-of-Experts (MoE) models scale more effectively than dense models due
  to sparse computation through expert routing, selectively activating only a
  small subset of expert modules. However, sparse computation challenges
  traditional training practices, as discrete expert routing hinders standard
  backpropagation and thus gradient-based optimization, which are the cornerstone
  of deep learning. To better pursue the scaling power of MoE, we introduce GRIN
  (GRadient-INformed MoE training), which incorporates sparse gradient estimation
  for expert routing and configures model parallelism to avoid token dropping.
  Applying GRIN to autoregressive language modeling, we develop a top-2
  16times3.8B MoE model. Our model, with only 6.6B activated parameters,
  outperforms a 7B dense model and matches the performance of a 14B dense model
  trained on the same data. Extensive evaluations across diverse tasks
  demonstrate the potential of GRIN to significantly enhance MoE efficacy,
  achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
paper_authors._id: 66eb8df368e69cde8f45fa81
paper_authors.name: Liyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:48:58.458Z
paper_authors._id: 6476372871d420a1f6be3650
paper_authors.avatarUrl: /avatars/ca3492d3967e06a7785457dd0ba2582c.svg
paper_authors.fullname: Liyuan Liu
paper_authors.type: user
paper_authors.user: LiyuanLucasLiu
paper_authors._id: 66eb8df368e69cde8f45fa82
paper_authors.name: Young Jin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:06:40.873Z
paper_authors._id: 63f5173bb51da4d61da6c038
paper_authors.avatarUrl: /avatars/0ee530cf80476aa3985c4d591cd384a1.svg
paper_authors.fullname: Young Jin Kim
paper_authors.type: user
paper_authors.user: ykim362
paper_authors._id: 66eb8df368e69cde8f45fa83
paper_authors.name: Shuohang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:49:08.370Z
paper_authors._id: 6463b2247572c66a8e625a57
paper_authors.avatarUrl: /avatars/7722fb5649d42d966ce1e478946d5f8f.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Shuohang
paper_authors._id: 66eb8df368e69cde8f45fa84
paper_authors.name: Chen Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:48:28.022Z
paper_authors._id: 63e6b5e22d2c508de9001afd
paper_authors.avatarUrl: /avatars/43cec9e8b8d490bd259e383954846a1e.svg
paper_authors.fullname: Chen Liang
paper_authors.type: user
paper_authors.user: cliang1453
paper_authors._id: 66eb8df368e69cde8f45fa85
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:49:48.130Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 66eb8df368e69cde8f45fa86
paper_authors.name: Hao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:48:30.010Z
paper_authors._id: 608c49299d7c9519b4adae25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/608c49299d7c9519b4adae25/pWSW52_QU7VD9hhDUHbmq.jpeg
paper_authors.fullname: Hao Cheng
paper_authors.type: user
paper_authors.user: kelvinih
paper_authors._id: 66eb8df368e69cde8f45fa87
paper_authors.name: Xiaodong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:50:29.011Z
paper_authors._id: 64b6007dfc5a8ae2a7e761ca
paper_authors.avatarUrl: /avatars/7d59d9e19ea536ce388a83764e48867d.svg
paper_authors.fullname: xiaodong liu
paper_authors.type: user
paper_authors.user: ArivnLiu
paper_authors._id: 66eb8df368e69cde8f45fa88
paper_authors.name: Masahiro Tanaka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:50:47.870Z
paper_authors._id: 65160d2c8a8269300db0a3b5
paper_authors.avatarUrl: /avatars/176d34200eca72791ad7f8393917513d.svg
paper_authors.fullname: Masahiro Tanaka
paper_authors.type: user
paper_authors.user: mtanaka
paper_authors._id: 66eb8df368e69cde8f45fa89
paper_authors.name: Xiaoxia Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:50:59.095Z
paper_authors._id: 66988ef14fe8c1a8acdac40b
paper_authors.avatarUrl: /avatars/611798a0c4532d7f61fd13ea8f685f38.svg
paper_authors.fullname: Xiaoxia Wu
paper_authors.type: user
paper_authors.user: xiaoxiawu123
paper_authors._id: 66eb8df368e69cde8f45fa8a
paper_authors.name: Wenxiang Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:51:11.468Z
paper_authors._id: 6327cafc46ffcfc27cfa4c9a
paper_authors.avatarUrl: /avatars/4daea27b432ff428c41d590669f7330c.svg
paper_authors.fullname: Wenxiang Hu
paper_authors.type: user
paper_authors.user: wenxcs
paper_authors._id: 66eb8df368e69cde8f45fa8b
paper_authors.name: Vishrav Chaudhary
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:51:31.110Z
paper_authors._id: 659c7ac977ac6f1bf5e63d7e
paper_authors.avatarUrl: /avatars/86a6efde0d483564a67ed5f344d479a0.svg
paper_authors.fullname: Vishrav Chaudhary
paper_authors.type: user
paper_authors.user: vishravmsft
paper_authors._id: 66eb8df368e69cde8f45fa8c
paper_authors.name: Zeqi Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:52:18.361Z
paper_authors._id: 642a862525442667e9ba5e78
paper_authors.avatarUrl: /avatars/f15bba13cb4096f6b3f3508ac2c56e5d.svg
paper_authors.fullname: Zeqi Lin
paper_authors.type: user
paper_authors.user: linzeqipku
paper_authors._id: 66eb8df368e69cde8f45fa8d
paper_authors.name: Chenruidong Zhang
paper_authors._id: 66eb8df368e69cde8f45fa8e
paper_authors.name: Jilong Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:52:32.876Z
paper_authors._id: 632bfc0e4a4991e7114f2a83
paper_authors.avatarUrl: /avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
paper_authors.fullname: Jilong Xue
paper_authors.type: user
paper_authors.user: jlxue
paper_authors._id: 66eb8df368e69cde8f45fa8f
paper_authors.name: Hany Awadalla
paper_authors._id: 66eb8df368e69cde8f45fa90
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:52:45.680Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 66eb8df368e69cde8f45fa91
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:52:51.947Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_title: GRIN: GRadient-INformed MoE
paper_summary: Mixture-of-Experts (MoE) models scale more effectively than dense models due
  to sparse computation through expert routing, selectively activating only a
  small subset of expert modules. However, sparse computation challenges
  traditional training practices, as discrete expert routing hinders standard
  backpropagation and thus gradient-based optimization, which are the cornerstone
  of deep learning. To better pursue the scaling power of MoE, we introduce GRIN
  (GRadient-INformed MoE training), which incorporates sparse gradient estimation
  for expert routing and configures model parallelism to avoid token dropping.
  Applying GRIN to autoregressive language modeling, we develop a top-2
  16times3.8B MoE model. Our model, with only 6.6B activated parameters,
  outperforms a 7B dense model and matches the performance of a 14B dense model
  trained on the same data. Extensive evaluations across diverse tasks
  demonstrate the potential of GRIN to significantly enhance MoE efficacy,
  achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
paper_ai_summary: GRIN enhances MoE models by using sparse gradient estimation for expert routing, enabling better scaling and performance compared to dense models.
paper_ai_keywords: Mixture-of-Experts MoE sparse computation expert routing gradient-based optimization GRIN autoregressive language modeling top-2 parameter-efficient MMLU HellaSwag HumanEval MATH