title: Layerwise Recurrent Router for Mixture-of-Experts
summary: The scaling of large language models (LLMs) has revolutionized their
  capabilities in various tasks, yet this growth must be matched with efficient
  computational strategies. The Mixture-of-Experts (MoE) architecture stands out
  for its ability to scale model size without significantly increasing training
  costs. Despite their advantages, current MoE models often display parameter
  inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion
  parameters might perform comparably to a standard model with 6.7 billion
  parameters. Being a crucial part of MoE, current routers in different layers
  independently assign tokens without leveraging historical routing information,
  potentially leading to suboptimal token-expert combinations and the parameter
  inefficiency problem. To alleviate this issue, we introduce the Layerwise
  Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated
  Recurrent Unit (GRU) to establish dependencies between routing decisions across
  consecutive layers. Such layerwise recurrence can be efficiently parallelly
  computed for input tokens and introduces negotiable costs. Our extensive
  empirical evaluations demonstrate that RMoE-based language models consistently
  outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel
  computation stage orthogonal to existing methods, allowing seamless
  compatibility with other MoE architectures. Our analyses attribute RMoE's gains
  to its effective cross-layer information sharing, which also improves expert
  selection and diversity. Our code is at https://github.com/qiuzh20/RMoE
paper_authors._id: 66bc2939181edd124e610e91
paper_authors.name: Zihan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:01:59.811Z
paper_authors._id: 647ccbd6e07cf9bb2d485244
paper_authors.avatarUrl: /avatars/e8915abaff04f6762247e196b7cf84df.svg
paper_authors.fullname: Zihan Qiu
paper_authors.type: user
paper_authors.user: QwQZh
paper_authors._id: 66bc2939181edd124e610e92
paper_authors.name: Zeyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T12:42:38.497Z
paper_authors._id: 64a5520dea3c5861303064b0
paper_authors.avatarUrl: /avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
paper_authors.fullname: Zeyu hUANG
paper_authors.type: user
paper_authors.user: zeroyhuang
paper_authors._id: 66bc2939181edd124e610e93
paper_authors.name: Shuang Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:11:36.552Z
paper_authors._id: 63478bc1e5c0717e673a086d
paper_authors.avatarUrl: /avatars/66513149f2e92a65bec54364c2d360c0.svg
paper_authors.fullname: Cheng Shuang
paper_authors.type: user
paper_authors.user: chengs18
paper_authors._id: 66bc2939181edd124e610e94
paper_authors.name: Yizhi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:02:47.766Z
paper_authors._id: 634182c506a304bdc43037c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665237671855-noauth.png
paper_authors.fullname: YIZHI ZHOU
paper_authors.type: user
paper_authors.user: IanZhou11
paper_authors._id: 66bc2939181edd124e610e95
paper_authors.name: Zili Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:03:10.383Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 66bc2939181edd124e610e96
paper_authors.name: Ivan Titov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:03:16.068Z
paper_authors._id: 666f6375623133f1ce79c021
paper_authors.avatarUrl: /avatars/b57a457cff81d685fdd9eadee53445fc.svg
paper_authors.fullname: Ivan Titov
paper_authors.type: user
paper_authors.user: Ivanchoo
paper_authors._id: 66bc2939181edd124e610e97
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T12:56:33.814Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: Layerwise Recurrent Router for Mixture-of-Experts
paper_summary: The scaling of large language models (LLMs) has revolutionized their
  capabilities in various tasks, yet this growth must be matched with efficient
  computational strategies. The Mixture-of-Experts (MoE) architecture stands out
  for its ability to scale model size without significantly increasing training
  costs. Despite their advantages, current MoE models often display parameter
  inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion
  parameters might perform comparably to a standard model with 6.7 billion
  parameters. Being a crucial part of MoE, current routers in different layers
  independently assign tokens without leveraging historical routing information,
  potentially leading to suboptimal token-expert combinations and the parameter
  inefficiency problem. To alleviate this issue, we introduce the Layerwise
  Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated
  Recurrent Unit (GRU) to establish dependencies between routing decisions across
  consecutive layers. Such layerwise recurrence can be efficiently parallelly
  computed for input tokens and introduces negotiable costs. Our extensive
  empirical evaluations demonstrate that RMoE-based language models consistently
  outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel
  computation stage orthogonal to existing methods, allowing seamless
  compatibility with other MoE architectures. Our analyses attribute RMoE's gains
  to its effective cross-layer information sharing, which also improves expert
  selection and diversity. Our code is at https://github.com/qiuzh20/RMoE
paper_ai_summary: Layerwise Recurrent Router for Mixture-of-Experts (RMoE) improves MoE models by leveraging GRU-based cross-layer dependencies, enhancing expert selection and diversity without significant computational cost.
paper_ai_keywords: Large language models (LLMs) Mixture-of-Experts (MoE) scaling parameter inefficiency routers Gated Recurrent Unit (GRU) Layerwise Recurrent Router for Mixture-of-Experts (RMoE) expert selection diversity