Yuan 2.0-M32: Mixture of Experts with Attention Router
Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
mixture-of-experts architecture with 32 experts of which 2 experts are active.
A new router network, Attention Router, is proposed and adopted for a more
efficient selection of experts, which boosts the accuracy of 3.8% compared to
the model with classical router network. Yuan 2.0-M32 is trained with 2000B
tokens from scratch, and the training computation consumption is only 9.25% of
a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
competitive capability on coding, math, and various domains of expertise, with
only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
respectively. The models and source codes of Yuan 2.0-M32 are released at
Github.
665698a71b4aa2651acc66b1
Shaohua Wu
665698a71b4aa2651acc66b2
Jiangang Luo
665698a71b4aa2651acc66b3
Xi Chen
admin_assigned
2024-05-29T12:50:31.646Z
6446572c3c72f182c94fdc29
/avatars/6e0fc34f46e47fcaf6267c50a5f4cbd7.svg
chenxi
user
chenxi0415
665698a71b4aa2651acc66b4
Lingjun Li
admin_assigned
2024-05-29T12:51:43.533Z
6465d2f7e48d80162317a15b
/avatars/ae80527357a0a5a25f251efc55667520.svg
Lingjun Li
user
Hico
665698a71b4aa2651acc66b5
Xudong Zhao
admin_assigned
2024-05-29T12:52:15.994Z
65671f8790aa29ce91c67e64
/avatars/1c037f2094834f2812fc6b7d40f5480e.svg
zhaoxudong
user
zhaoxudong01
665698a71b4aa2651acc66b6
Tong Yu
665698a71b4aa2651acc66b7
Chao Wang
665698a71b4aa2651acc66b8
Yue Wang
665698a71b4aa2651acc66b9
Fei Wang
665698a71b4aa2651acc66ba
Weixu Qiao
665698a71b4aa2651acc66bb
Houbo He
admin_assigned
2024-05-29T12:47:41.076Z
64c220da91094b0c335b65e3
/avatars/af0c50ad503cbf1edc4b58552cb86ea8.svg
hehoubo
user
hehoubo
665698a71b4aa2651acc66bc
Zeru Zhang
admin_assigned
2024-05-29T12:51:12.503Z
64c9cb40b0e612877895ca7d
/avatars/aeabb1333a72635296f8b55e85ef7e16.svg
zhangzeru
user
summyze
665698a71b4aa2651acc66bd
Zeyu Sun
admin_assigned
2024-05-29T12:47:29.622Z
663efbcd663bb02184b9e542
/avatars/6896c411cb6297fcb75bd3f36e0a1805.svg
Zeyu Sun
user
zestyzapus
665698a71b4aa2651acc66be
Junxiong Mao
665698a71b4aa2651acc66bf
Chong Shen
Yuan 2.0-M32: Mixture of Experts with Attention Router
Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
mixture-of-experts architecture with 32 experts of which 2 experts are active.
A new router network, Attention Router, is proposed and adopted for a more
efficient selection of experts, which boosts the accuracy of 3.8% compared to
the model with classical router network. Yuan 2.0-M32 is trained with 2000B
tokens from scratch, and the training computation consumption is only 9.25% of
a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
competitive capability on coding, math, and various domains of expertise, with
only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
respectively. The models and source codes of Yuan 2.0-M32 are released at
Github.
Yuan 2.0-M32, using a mixture-of-experts architecture with an Attention Router, surpasses Llama3-70B on MATH and ARC-Challenge benchmarks with lower computational requirements and parameters.
mixture-of-experts architecture Attention Router forward computation per token Llama3-70B MATH benchmark ARC-Challenge benchmark