title: A Closer Look into Mixture-of-Experts in Large Language Models
summary: Mixture-of-experts (MoE) is gaining increasing attention due to its unique
  properties and remarkable performance, especially for language tasks. By
  sparsely activating a subset of parameters for each token, MoE architecture
  could increase the model size without sacrificing computational efficiency,
  achieving a better trade-off between performance and training costs. However,
  the underlying mechanism of MoE still lacks further exploration, and its
  modularization degree remains questionable. In this paper, we make an initial
  attempt to understand the inner workings of MoE-based large language models.
  Concretely, we comprehensively study the parametric and behavioral features of
  three recent MoE-based models and reveal some intriguing observations,
  including (1) Neurons act like fine-grained experts. (2) The router of MoE
  usually selects experts with larger output norms. (3) The expert diversity
  increases as the layer increases, while the last layer is an outlier. Based on
  the observations, we also provide suggestions for a broad spectrum of MoE
  practitioners, such as router design and expert allocation. We hope this work
  could shed light on future research on the MoE framework and other modular
  architectures. Code is available at
  https://github.com/kamanphoebe/Look-into-MoEs.
paper_authors._id: 667cdd96eb164a3fe04ca8c4
paper_authors.name: Ka Man Lo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T07:31:03.371Z
paper_authors._id: 649ad817c5d076065a514a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ad817c5d076065a514a41/WsGymqaiZFsx3jDtMCycl.jpeg
paper_authors.fullname: Ka Man Lo
paper_authors.type: user
paper_authors.user: kamanphoebe
paper_authors._id: 667cdd96eb164a3fe04ca8c5
paper_authors.name: Zeyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T12:42:37.238Z
paper_authors._id: 64a5520dea3c5861303064b0
paper_authors.avatarUrl: /avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
paper_authors.fullname: Zeyu hUANG
paper_authors.type: user
paper_authors.user: zeroyhuang
paper_authors._id: 667cdd96eb164a3fe04ca8c6
paper_authors.name: Zihan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T15:38:00.251Z
paper_authors._id: 647ccbd6e07cf9bb2d485244
paper_authors.avatarUrl: /avatars/e8915abaff04f6762247e196b7cf84df.svg
paper_authors.fullname: Zihan Qiu
paper_authors.type: user
paper_authors.user: QwQZh
paper_authors._id: 667cdd96eb164a3fe04ca8c7
paper_authors.name: Zili Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T15:38:06.551Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 667cdd96eb164a3fe04ca8c8
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T13:44:54.008Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: A Closer Look into Mixture-of-Experts in Large Language Models
paper_summary: Mixture-of-experts (MoE) is gaining increasing attention due to its unique
  properties and remarkable performance, especially for language tasks. By
  sparsely activating a subset of parameters for each token, MoE architecture
  could increase the model size without sacrificing computational efficiency,
  achieving a better trade-off between performance and training costs. However,
  the underlying mechanism of MoE still lacks further exploration, and its
  modularization degree remains questionable. In this paper, we make an initial
  attempt to understand the inner workings of MoE-based large language models.
  Concretely, we comprehensively study the parametric and behavioral features of
  three recent MoE-based models and reveal some intriguing observations,
  including (1) Neurons act like fine-grained experts. (2) The router of MoE
  usually selects experts with larger output norms. (3) The expert diversity
  increases as the layer increases, while the last layer is an outlier. Based on
  the observations, we also provide suggestions for a broad spectrum of MoE
  practitioners, such as router design and expert allocation. We hope this work
  could shed light on future research on the MoE framework and other modular
  architectures. Code is available at
  https://github.com/kamanphoebe/Look-into-MoEs.
paper_ai_keywords: mixture-of-experts (MoE) sparsely activating computational efficiency parametric features behavioral features neurons fine-grained experts router output norms expert diversity modularization degree router design expert allocation