Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
  Parameters by Tencent
In this paper, we introduce Hunyuan-Large, which is currently the largest
open-source Transformer-based mixture of experts model, with a total of 389
billion parameters and 52 billion activation parameters, capable of handling up
to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior
performance across various benchmarks including language understanding and
generation, logical reasoning, mathematical problem-solving, coding,
long-context, and aggregated tasks, where it outperforms LLama3.1-70B and
exhibits comparable performance when compared to the significantly larger
LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale
synthetic data that is orders larger than in previous literature, a mixed
expert routing strategy, a key-value cache compression technique, and an
expert-specific learning rate strategy. Additionally, we also investigate the
scaling laws and learning rate schedule of mixture of experts models, providing
valuable insights and guidances for future model development and optimization.
The code and checkpoints of Hunyuan-Large are released to facilitate future
innovations and applications.
  Codes: https://github.com/Tencent/Hunyuan-Large
  Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
67298ecccdd83d27791f1dc9
Xingwu Sun
67298ecccdd83d27791f1dca
Yanfeng Chen
admin_assigned
2024-11-05T08:03:55.794Z
64c99c82bd5f839f2fdeb173
/avatars/194b73285cef5e6e71338cafaae48eeb.svg
Yanfeng Chen
user
YanfengChen
67298ecccdd83d27791f1dcb
Yiqing Huang
admin_assigned
2024-11-05T08:49:02.756Z
64f81e1f2003abb61b09eed1
/avatars/e6282a815e44fb355b53a9a99a56fe88.svg
Yiqing Huang
user
Mimosa77
67298ecccdd83d27791f1dcc
Ruobing Xie
admin_assigned
2024-11-05T08:49:11.945Z
6622443b9b0614a760dd8123
/avatars/acb6c1c9c429af1112530dcf76a8e420.svg
Ruobing Xie
user
Ruobing-Xie
67298ecccdd83d27791f1dcd
Jiaqi Zhu
67298ecccdd83d27791f1dce
Kai Zhang
67298ecccdd83d27791f1dcf
Shuaipeng Li
claimed_verified
2025-01-08T09:48:20.228Z
64e32fb7e5bd41dd056bc943
/avatars/fbe84d0641dfdd665cffb1aad464e8a0.svg
Shuaipeng Li
user
unlimblue
67298ecccdd83d27791f1dd0
Zhen Yang
claimed_verified
2025-01-09T10:11:31.413Z
62c4057732fa66fedacca0db
/avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
Zhen Yang
user
andyyang
67298ecccdd83d27791f1dd1
Jonny Han
67298ecccdd83d27791f1dd2
Xiaobo Shu
67298ecccdd83d27791f1dd3
Jiahao Bu
67298ecccdd83d27791f1dd4
Zhongzhi Chen
67298ecccdd83d27791f1dd5
Xuemeng Huang
67298ecccdd83d27791f1dd6
Fengzong Lian
67298ecccdd83d27791f1dd7
Saiyong Yang
67298ecccdd83d27791f1dd8
Jianfeng Yan
67298ecccdd83d27791f1dd9
Yuyuan Zeng
67298ecccdd83d27791f1dda
Xiaoqin Ren
67298ecccdd83d27791f1ddb
Chao Yu
67298ecccdd83d27791f1ddc
Lulu Wu
67298ecccdd83d27791f1ddd
Yue Mao
67298ecccdd83d27791f1dde
Tao Yang
67298ecccdd83d27791f1ddf
Suncong Zheng
67298ecccdd83d27791f1de0
Kan Wu
67298ecccdd83d27791f1de1
Dian Jiao
67298ecccdd83d27791f1de2
Jinbao Xue
67298ecccdd83d27791f1de3
Xipeng Zhang
67298ecccdd83d27791f1de4
Decheng Wu
67298ecccdd83d27791f1de5
Kai Liu
67298ecccdd83d27791f1de6
Dengpeng Wu
67298ecccdd83d27791f1de7
Guanghui Xu
67298ecccdd83d27791f1de8
Shaohua Chen
67298ecccdd83d27791f1de9
Shuang Chen
67298ecccdd83d27791f1dea
Xiao Feng
67298ecccdd83d27791f1deb
Yigeng Hong
67298ecccdd83d27791f1dec
Junqiang Zheng
67298ecccdd83d27791f1ded
Chengcheng Xu
67298ecccdd83d27791f1dee
Zongwei Li
67298ecccdd83d27791f1def
Xiong Kuang
67298ecccdd83d27791f1df0
Jianglu Hu
67298ecccdd83d27791f1df1
Yiqi Chen
67298ecccdd83d27791f1df2
Yuchi Deng
67298ecccdd83d27791f1df3
Guiyang Li
67298ecccdd83d27791f1df4
Ao Liu
67298ecccdd83d27791f1df5
Chenchen Zhang
claimed_verified
2024-11-05T13:36:26.125Z
64b74b906ab5d14ca7f289cd
/avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
xxzcc
user
xxzcc
67298ecccdd83d27791f1df6
Shihui Hu
67298ecccdd83d27791f1df7
Zilong Zhao
67298ecccdd83d27791f1df8
Zifan Wu
67298ecccdd83d27791f1df9
Yao Ding
67298ecccdd83d27791f1dfa
Weichao Wang
67298ecccdd83d27791f1dfb
Han Liu
67298ecccdd83d27791f1dfc
Roberts Wang
67298ecccdd83d27791f1dfd
Hao Fei
67298ecccdd83d27791f1dfe
Peijie She
67298ecccdd83d27791f1dff
Ze Zhao
67298ecccdd83d27791f1e00
Xun Cao
67298ecccdd83d27791f1e01
Hai Wang
67298ecccdd83d27791f1e02
Fusheng Xiang
67298ecccdd83d27791f1e03
Mengyuan Huang
67298ecccdd83d27791f1e04
Zhiyuan Xiong
67298ecccdd83d27791f1e05
Bin Hu
67298ecccdd83d27791f1e06
Xuebin Hou
67298ecccdd83d27791f1e07
Lei Jiang
67298ecccdd83d27791f1e08
Jiajia Wu
67298ecccdd83d27791f1e09
Yaping Deng
67298ecccdd83d27791f1e0a
Yi Shen
67298ecccdd83d27791f1e0b
Qian Wang
67298ecccdd83d27791f1e0c
Weijie Liu
67298ecccdd83d27791f1e0d
Jie Liu
67298ecccdd83d27791f1e0e
Meng Chen
67298ecccdd83d27791f1e0f
Liang Dong
67298ecccdd83d27791f1e10
Weiwen Jia
67298ecccdd83d27791f1e11
Hu Chen
67298ecccdd83d27791f1e12
Feifei Liu
67298ecccdd83d27791f1e13
Rui Yuan
67298ecccdd83d27791f1e14
Huilin Xu
67298ecccdd83d27791f1e15
Zhenxiang Yan
67298ecccdd83d27791f1e16
Tengfei Cao
67298ecccdd83d27791f1e17
Zhichao Hu
67298ecccdd83d27791f1e18
Xinhua Feng
67298ecccdd83d27791f1e19
Dong Du
claimed_verified
2025-07-29T07:13:13.971Z
651eaaad567de56557e7bde6
/avatars/7ddf08fe4459fa2b7b3796be4eb4f43d.svg
duu
user
dongdongdongdu
67298ecccdd83d27791f1e1a
Tinghao She
67298ecccdd83d27791f1e1b
Yangyu Tao
67298ecccdd83d27791f1e1c
Feng Zhang
67298ecccdd83d27791f1e1d
Jianchen Zhu
67298ecccdd83d27791f1e1e
Chengzhong Xu
67298ecccdd83d27791f1e1f
Xirui Li
67298ecccdd83d27791f1e20
Chong Zha
67298ecccdd83d27791f1e21
Wen Ouyang
67298ecccdd83d27791f1e22
Yinben Xia
67298ecccdd83d27791f1e23
Xiang Li
67298ecccdd83d27791f1e24
Zekun He
67298ecccdd83d27791f1e25
Rongpeng Chen
67298ecccdd83d27791f1e26
Jiawei Song
67298ecccdd83d27791f1e27
Ruibin Chen
67298ecccdd83d27791f1e28
Fan Jiang
67298ecccdd83d27791f1e29
Chongqing Zhao
67298ecccdd83d27791f1e2a
Bo Wang
67298ecccdd83d27791f1e2b
Hao Gong
67298ecccdd83d27791f1e2c
Rong Gan
67298ecccdd83d27791f1e2d
Winston Hu
67298ecccdd83d27791f1e2e
Zhanhui Kang
67298ecccdd83d27791f1e2f
Yong Yang
67298ecccdd83d27791f1e30
Yuhong Liu
67298ecccdd83d27791f1e31
Di Wang
67298ecccdd83d27791f1e32
Jie Jiang
Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
  Parameters by Tencent
In this paper, we introduce Hunyuan-Large, which is currently the largest
open-source Transformer-based mixture of experts model, with a total of 389
billion parameters and 52 billion activation parameters, capable of handling up
to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior
performance across various benchmarks including language understanding and
generation, logical reasoning, mathematical problem-solving, coding,
long-context, and aggregated tasks, where it outperforms LLama3.1-70B and
exhibits comparable performance when compared to the significantly larger
LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale
synthetic data that is orders larger than in previous literature, a mixed
expert routing strategy, a key-value cache compression technique, and an
expert-specific learning rate strategy. Additionally, we also investigate the
scaling laws and learning rate schedule of mixture of experts models, providing
valuable insights and guidances for future model development and optimization.
The code and checkpoints of Hunyuan-Large are released to facilitate future
innovations and applications.
  Codes: https://github.com/Tencent/Hunyuan-Large
  Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
Hunyuan-Large, a large-scale Transformer-based mixture of experts model, outperforms smaller models across various benchmarks and provides insights into the scaling laws and learning rates for such models.
Transformer-based mixture of experts Hunyuan-Large parameters activation parameters tokens language understanding language generation logical reasoning mathematical problem-solving coding long-context tasks aggregated tasks LLama3.1-70B LLama3.1-405B large-scale synthetic data mixed expert routing strategy key-value cache compression expert-specific learning rate strategy scaling laws learning rate schedule