title: Autonomy-of-Experts Models
summary: Mixture-of-Experts (MoE) models mostly use a router to assign tokens to
  specific expert modules, activating only partial parameters and often
  outperforming dense models. We argue that the separation between the router's
  decision-making and the experts' execution is a critical yet overlooked issue,
  leading to suboptimal expert selection and ineffective learning. To address
  this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which
  experts autonomously select themselves to process inputs. AoE is based on the
  insight that an expert is aware of its own capacity to effectively process a
  token, an awareness reflected in the scale of its internal activations. In AoE,
  routers are removed; instead, experts pre-compute internal activations for
  inputs and are ranked based on their activation norms. Only the top-ranking
  experts proceed with the forward pass, while the others abort. The overhead of
  pre-computing activations is reduced through a low-rank weight factorization.
  This self-evaluating-then-partner-comparing approach ensures improved expert
  selection and effective learning. We pre-train language models having 700M up
  to 4B parameters, demonstrating that AoE outperforms traditional MoE models
  with comparable efficiency.
paper_authors._id: 6791baa69d6eba7f285d5dce
paper_authors.name: Ang Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:39:56.484Z
paper_authors._id: 64b8ca3c5067873176d4b436
paper_authors.avatarUrl: /avatars/b659d147b2454b47c9a7e89bbed525fc.svg
paper_authors.fullname: AngLv
paper_authors.type: user
paper_authors.user: AngLv
paper_authors._id: 6791baa69d6eba7f285d5dcf
paper_authors.name: Ruobing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:40:02.493Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 6791baa69d6eba7f285d5dd0
paper_authors.name: Yining Qian
paper_authors._id: 6791baa69d6eba7f285d5dd1
paper_authors.name: Songhao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:40:22.474Z
paper_authors._id: 662aa42f4eaa187e4cf6827b
paper_authors.avatarUrl: /avatars/17139f0b6e8092cf4c135028db03a7ff.svg
paper_authors.fullname: Songhao Wu
paper_authors.type: user
paper_authors.user: shwu
paper_authors._id: 6791baa69d6eba7f285d5dd2
paper_authors.name: Xingwu Sun
paper_authors._id: 6791baa69d6eba7f285d5dd3
paper_authors.name: Zhanhui Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:40:42.332Z
paper_authors._id: 6728117b71b9baba45c25c35
paper_authors.avatarUrl: /avatars/5e4603f00a426c6c41e3e6fef5fa0362.svg
paper_authors.fullname: zhanhui kang
paper_authors.type: user
paper_authors.user: kangzhanhui
paper_authors._id: 6791baa69d6eba7f285d5dd4
paper_authors.name: Di Wang
paper_authors._id: 6791baa69d6eba7f285d5dd5
paper_authors.name: Rui Yan
paper_title: Autonomy-of-Experts Models
paper_summary: Mixture-of-Experts (MoE) models mostly use a router to assign tokens to
  specific expert modules, activating only partial parameters and often
  outperforming dense models. We argue that the separation between the router's
  decision-making and the experts' execution is a critical yet overlooked issue,
  leading to suboptimal expert selection and ineffective learning. To address
  this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which
  experts autonomously select themselves to process inputs. AoE is based on the
  insight that an expert is aware of its own capacity to effectively process a
  token, an awareness reflected in the scale of its internal activations. In AoE,
  routers are removed; instead, experts pre-compute internal activations for
  inputs and are ranked based on their activation norms. Only the top-ranking
  experts proceed with the forward pass, while the others abort. The overhead of
  pre-computing activations is reduced through a low-rank weight factorization.
  This self-evaluating-then-partner-comparing approach ensures improved expert
  selection and effective learning. We pre-train language models having 700M up
  to 4B parameters, demonstrating that AoE outperforms traditional MoE models
  with comparable efficiency.
paper_ai_summary: Autonomy-of-Experts (AoE) improves MoE models by enabling experts to autonomously select inputs based on self-evaluated capacity, reducing the need for routers and enhancing performance.
paper_ai_keywords: Mixture-of-Experts MoE Autonomy-of-Experts AoE router expert modules parameter-efficient dense models internal activations activation norms low-rank weight factorization language models pre-training