title: Yuan 2.0-M32: Mixture of Experts with Attention Router
summary: Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
  mixture-of-experts architecture with 32 experts of which 2 experts are active.
  A new router network, Attention Router, is proposed and adopted for a more
  efficient selection of experts, which boosts the accuracy of 3.8% compared to
  the model with classical router network. Yuan 2.0-M32 is trained with 2000B
  tokens from scratch, and the training computation consumption is only 9.25% of
  a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
  competitive capability on coding, math, and various domains of expertise, with
  only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
  per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
  Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
  respectively. The models and source codes of Yuan 2.0-M32 are released at
  Github.
paper_authors._id: 665698a71b4aa2651acc66b1
paper_authors.name: Shaohua Wu
paper_authors._id: 665698a71b4aa2651acc66b2
paper_authors.name: Jiangang Luo
paper_authors._id: 665698a71b4aa2651acc66b3
paper_authors.name: Xi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:50:31.646Z
paper_authors._id: 6446572c3c72f182c94fdc29
paper_authors.avatarUrl: /avatars/6e0fc34f46e47fcaf6267c50a5f4cbd7.svg
paper_authors.fullname: chenxi
paper_authors.type: user
paper_authors.user: chenxi0415
paper_authors._id: 665698a71b4aa2651acc66b4
paper_authors.name: Lingjun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:51:43.533Z
paper_authors._id: 6465d2f7e48d80162317a15b
paper_authors.avatarUrl: /avatars/ae80527357a0a5a25f251efc55667520.svg
paper_authors.fullname: Lingjun Li
paper_authors.type: user
paper_authors.user: Hico
paper_authors._id: 665698a71b4aa2651acc66b5
paper_authors.name: Xudong Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:52:15.994Z
paper_authors._id: 65671f8790aa29ce91c67e64
paper_authors.avatarUrl: /avatars/1c037f2094834f2812fc6b7d40f5480e.svg
paper_authors.fullname: zhaoxudong
paper_authors.type: user
paper_authors.user: zhaoxudong01
paper_authors._id: 665698a71b4aa2651acc66b6
paper_authors.name: Tong Yu
paper_authors._id: 665698a71b4aa2651acc66b7
paper_authors.name: Chao Wang
paper_authors._id: 665698a71b4aa2651acc66b8
paper_authors.name: Yue Wang
paper_authors._id: 665698a71b4aa2651acc66b9
paper_authors.name: Fei Wang
paper_authors._id: 665698a71b4aa2651acc66ba
paper_authors.name: Weixu Qiao
paper_authors._id: 665698a71b4aa2651acc66bb
paper_authors.name: Houbo He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:47:41.076Z
paper_authors._id: 64c220da91094b0c335b65e3
paper_authors.avatarUrl: /avatars/af0c50ad503cbf1edc4b58552cb86ea8.svg
paper_authors.fullname: hehoubo
paper_authors.type: user
paper_authors.user: hehoubo
paper_authors._id: 665698a71b4aa2651acc66bc
paper_authors.name: Zeru Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:51:12.503Z
paper_authors._id: 64c9cb40b0e612877895ca7d
paper_authors.avatarUrl: /avatars/aeabb1333a72635296f8b55e85ef7e16.svg
paper_authors.fullname: zhangzeru
paper_authors.type: user
paper_authors.user: summyze
paper_authors._id: 665698a71b4aa2651acc66bd
paper_authors.name: Zeyu Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-29T12:47:29.622Z
paper_authors._id: 663efbcd663bb02184b9e542
paper_authors.avatarUrl: /avatars/6896c411cb6297fcb75bd3f36e0a1805.svg
paper_authors.fullname: Zeyu Sun
paper_authors.type: user
paper_authors.user: zestyzapus
paper_authors._id: 665698a71b4aa2651acc66be
paper_authors.name: Junxiong Mao
paper_authors._id: 665698a71b4aa2651acc66bf
paper_authors.name: Chong Shen
paper_title: Yuan 2.0-M32: Mixture of Experts with Attention Router
paper_summary: Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
  mixture-of-experts architecture with 32 experts of which 2 experts are active.
  A new router network, Attention Router, is proposed and adopted for a more
  efficient selection of experts, which boosts the accuracy of 3.8% compared to
  the model with classical router network. Yuan 2.0-M32 is trained with 2000B
  tokens from scratch, and the training computation consumption is only 9.25% of
  a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
  competitive capability on coding, math, and various domains of expertise, with
  only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
  per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
  Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
  respectively. The models and source codes of Yuan 2.0-M32 are released at
  Github.
paper_ai_keywords: mixture-of-experts architecture experts Attention Router router network efficient selection accuracy training computation consumption dense model forward computation MATH benchmark ARC-Challenge benchmark