title: Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
    Parameters by Tencent
summary: In this paper, we introduce Hunyuan-Large, which is currently the largest
  open-source Transformer-based mixture of experts model, with a total of 389
  billion parameters and 52 billion activation parameters, capable of handling up
  to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior
  performance across various benchmarks including language understanding and
  generation, logical reasoning, mathematical problem-solving, coding,
  long-context, and aggregated tasks, where it outperforms LLama3.1-70B and
  exhibits comparable performance when compared to the significantly larger
  LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale
  synthetic data that is orders larger than in previous literature, a mixed
  expert routing strategy, a key-value cache compression technique, and an
  expert-specific learning rate strategy. Additionally, we also investigate the
  scaling laws and learning rate schedule of mixture of experts models, providing
  valuable insights and guidances for future model development and optimization.
  The code and checkpoints of Hunyuan-Large are released to facilitate future
  innovations and applications.
    Codes: https://github.com/Tencent/Hunyuan-Large
    Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
paper_authors._id: 67298ecccdd83d27791f1dc9
paper_authors.name: Xingwu Sun
paper_authors._id: 67298ecccdd83d27791f1dca
paper_authors.name: Yanfeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T08:03:55.794Z
paper_authors._id: 64c99c82bd5f839f2fdeb173
paper_authors.avatarUrl: /avatars/194b73285cef5e6e71338cafaae48eeb.svg
paper_authors.fullname: Yanfeng Chen
paper_authors.type: user
paper_authors.user: YanfengChen
paper_authors._id: 67298ecccdd83d27791f1dcb
paper_authors.name: Yiqing Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T08:49:02.756Z
paper_authors._id: 64f81e1f2003abb61b09eed1
paper_authors.avatarUrl: /avatars/e6282a815e44fb355b53a9a99a56fe88.svg
paper_authors.fullname: Yiqing Huang
paper_authors.type: user
paper_authors.user: Mimosa77
paper_authors._id: 67298ecccdd83d27791f1dcc
paper_authors.name: Ruobing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T08:49:11.945Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 67298ecccdd83d27791f1dcd
paper_authors.name: Jiaqi Zhu
paper_authors._id: 67298ecccdd83d27791f1dce
paper_authors.name: Kai Zhang
paper_authors._id: 67298ecccdd83d27791f1dcf
paper_authors.name: Shuaipeng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:48:20.228Z
paper_authors._id: 64e32fb7e5bd41dd056bc943
paper_authors.avatarUrl: /avatars/fbe84d0641dfdd665cffb1aad464e8a0.svg
paper_authors.fullname: Shuaipeng Li
paper_authors.type: user
paper_authors.user: unlimblue
paper_authors._id: 67298ecccdd83d27791f1dd0
paper_authors.name: Zhen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:11:31.413Z
paper_authors._id: 62c4057732fa66fedacca0db
paper_authors.avatarUrl: /avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
paper_authors.fullname: AndyYang
paper_authors.type: user
paper_authors.user: andyyang
paper_authors._id: 67298ecccdd83d27791f1dd1
paper_authors.name: Jonny Han
paper_authors._id: 67298ecccdd83d27791f1dd2
paper_authors.name: Xiaobo Shu
paper_authors._id: 67298ecccdd83d27791f1dd3
paper_authors.name: Jiahao Bu
paper_authors._id: 67298ecccdd83d27791f1dd4
paper_authors.name: Zhongzhi Chen
paper_authors._id: 67298ecccdd83d27791f1dd5
paper_authors.name: Xuemeng Huang
paper_authors._id: 67298ecccdd83d27791f1dd6
paper_authors.name: Fengzong Lian
paper_authors._id: 67298ecccdd83d27791f1dd7
paper_authors.name: Saiyong Yang
paper_authors._id: 67298ecccdd83d27791f1dd8
paper_authors.name: Jianfeng Yan
paper_authors._id: 67298ecccdd83d27791f1dd9
paper_authors.name: Yuyuan Zeng
paper_authors._id: 67298ecccdd83d27791f1dda
paper_authors.name: Xiaoqin Ren
paper_authors._id: 67298ecccdd83d27791f1ddb
paper_authors.name: Chao Yu
paper_authors._id: 67298ecccdd83d27791f1ddc
paper_authors.name: Lulu Wu
paper_authors._id: 67298ecccdd83d27791f1ddd
paper_authors.name: Yue Mao
paper_authors._id: 67298ecccdd83d27791f1dde
paper_authors.name: Tao Yang
paper_authors._id: 67298ecccdd83d27791f1ddf
paper_authors.name: Suncong Zheng
paper_authors._id: 67298ecccdd83d27791f1de0
paper_authors.name: Kan Wu
paper_authors._id: 67298ecccdd83d27791f1de1
paper_authors.name: Dian Jiao
paper_authors._id: 67298ecccdd83d27791f1de2
paper_authors.name: Jinbao Xue
paper_authors._id: 67298ecccdd83d27791f1de3
paper_authors.name: Xipeng Zhang
paper_authors._id: 67298ecccdd83d27791f1de4
paper_authors.name: Decheng Wu
paper_authors._id: 67298ecccdd83d27791f1de5
paper_authors.name: Kai Liu
paper_authors._id: 67298ecccdd83d27791f1de6
paper_authors.name: Dengpeng Wu
paper_authors._id: 67298ecccdd83d27791f1de7
paper_authors.name: Guanghui Xu
paper_authors._id: 67298ecccdd83d27791f1de8
paper_authors.name: Shaohua Chen
paper_authors._id: 67298ecccdd83d27791f1de9
paper_authors.name: Shuang Chen
paper_authors._id: 67298ecccdd83d27791f1dea
paper_authors.name: Xiao Feng
paper_authors._id: 67298ecccdd83d27791f1deb
paper_authors.name: Yigeng Hong
paper_authors._id: 67298ecccdd83d27791f1dec
paper_authors.name: Junqiang Zheng
paper_authors._id: 67298ecccdd83d27791f1ded
paper_authors.name: Chengcheng Xu
paper_authors._id: 67298ecccdd83d27791f1dee
paper_authors.name: Zongwei Li
paper_authors._id: 67298ecccdd83d27791f1def
paper_authors.name: Xiong Kuang
paper_authors._id: 67298ecccdd83d27791f1df0
paper_authors.name: Jianglu Hu
paper_authors._id: 67298ecccdd83d27791f1df1
paper_authors.name: Yiqi Chen
paper_authors._id: 67298ecccdd83d27791f1df2
paper_authors.name: Yuchi Deng
paper_authors._id: 67298ecccdd83d27791f1df3
paper_authors.name: Guiyang Li
paper_authors._id: 67298ecccdd83d27791f1df4
paper_authors.name: Ao Liu
paper_authors._id: 67298ecccdd83d27791f1df5
paper_authors.name: Chenchen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T13:36:26.125Z
paper_authors._id: 64b74b906ab5d14ca7f289cd
paper_authors.avatarUrl: /avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
paper_authors.fullname: xxzcc
paper_authors.type: user
paper_authors.user: xxzcc
paper_authors._id: 67298ecccdd83d27791f1df6
paper_authors.name: Shihui Hu
paper_authors._id: 67298ecccdd83d27791f1df7
paper_authors.name: Zilong Zhao
paper_authors._id: 67298ecccdd83d27791f1df8
paper_authors.name: Zifan Wu
paper_authors._id: 67298ecccdd83d27791f1df9
paper_authors.name: Yao Ding
paper_authors._id: 67298ecccdd83d27791f1dfa
paper_authors.name: Weichao Wang
paper_authors._id: 67298ecccdd83d27791f1dfb
paper_authors.name: Han Liu
paper_authors._id: 67298ecccdd83d27791f1dfc
paper_authors.name: Roberts Wang
paper_authors._id: 67298ecccdd83d27791f1dfd
paper_authors.name: Hao Fei
paper_authors._id: 67298ecccdd83d27791f1dfe
paper_authors.name: Peijie She
paper_authors._id: 67298ecccdd83d27791f1dff
paper_authors.name: Ze Zhao
paper_authors._id: 67298ecccdd83d27791f1e00
paper_authors.name: Xun Cao
paper_authors._id: 67298ecccdd83d27791f1e01
paper_authors.name: Hai Wang
paper_authors._id: 67298ecccdd83d27791f1e02
paper_authors.name: Fusheng Xiang
paper_authors._id: 67298ecccdd83d27791f1e03
paper_authors.name: Mengyuan Huang
paper_authors._id: 67298ecccdd83d27791f1e04
paper_authors.name: Zhiyuan Xiong
paper_authors._id: 67298ecccdd83d27791f1e05
paper_authors.name: Bin Hu
paper_authors._id: 67298ecccdd83d27791f1e06
paper_authors.name: Xuebin Hou
paper_authors._id: 67298ecccdd83d27791f1e07
paper_authors.name: Lei Jiang
paper_authors._id: 67298ecccdd83d27791f1e08
paper_authors.name: Jiajia Wu
paper_authors._id: 67298ecccdd83d27791f1e09
paper_authors.name: Yaping Deng
paper_authors._id: 67298ecccdd83d27791f1e0a
paper_authors.name: Yi Shen
paper_authors._id: 67298ecccdd83d27791f1e0b
paper_authors.name: Qian Wang
paper_authors._id: 67298ecccdd83d27791f1e0c
paper_authors.name: Weijie Liu
paper_authors._id: 67298ecccdd83d27791f1e0d
paper_authors.name: Jie Liu
paper_authors._id: 67298ecccdd83d27791f1e0e
paper_authors.name: Meng Chen
paper_authors._id: 67298ecccdd83d27791f1e0f
paper_authors.name: Liang Dong
paper_authors._id: 67298ecccdd83d27791f1e10
paper_authors.name: Weiwen Jia
paper_authors._id: 67298ecccdd83d27791f1e11
paper_authors.name: Hu Chen
paper_authors._id: 67298ecccdd83d27791f1e12
paper_authors.name: Feifei Liu
paper_authors._id: 67298ecccdd83d27791f1e13
paper_authors.name: Rui Yuan
paper_authors._id: 67298ecccdd83d27791f1e14
paper_authors.name: Huilin Xu
paper_authors._id: 67298ecccdd83d27791f1e15
paper_authors.name: Zhenxiang Yan
paper_authors._id: 67298ecccdd83d27791f1e16
paper_authors.name: Tengfei Cao
paper_authors._id: 67298ecccdd83d27791f1e17
paper_authors.name: Zhichao Hu
paper_authors._id: 67298ecccdd83d27791f1e18
paper_authors.name: Xinhua Feng
paper_authors._id: 67298ecccdd83d27791f1e19
paper_authors.name: Dong Du
paper_authors._id: 67298ecccdd83d27791f1e1a
paper_authors.name: Tinghao She
paper_authors._id: 67298ecccdd83d27791f1e1b
paper_authors.name: Yangyu Tao
paper_authors._id: 67298ecccdd83d27791f1e1c
paper_authors.name: Feng Zhang
paper_authors._id: 67298ecccdd83d27791f1e1d
paper_authors.name: Jianchen Zhu
paper_authors._id: 67298ecccdd83d27791f1e1e
paper_authors.name: Chengzhong Xu
paper_authors._id: 67298ecccdd83d27791f1e1f
paper_authors.name: Xirui Li
paper_authors._id: 67298ecccdd83d27791f1e20
paper_authors.name: Chong Zha
paper_authors._id: 67298ecccdd83d27791f1e21
paper_authors.name: Wen Ouyang
paper_authors._id: 67298ecccdd83d27791f1e22
paper_authors.name: Yinben Xia
paper_authors._id: 67298ecccdd83d27791f1e23
paper_authors.name: Xiang Li
paper_authors._id: 67298ecccdd83d27791f1e24
paper_authors.name: Zekun He
paper_authors._id: 67298ecccdd83d27791f1e25
paper_authors.name: Rongpeng Chen
paper_authors._id: 67298ecccdd83d27791f1e26
paper_authors.name: Jiawei Song
paper_authors._id: 67298ecccdd83d27791f1e27
paper_authors.name: Ruibin Chen
paper_authors._id: 67298ecccdd83d27791f1e28
paper_authors.name: Fan Jiang
paper_authors._id: 67298ecccdd83d27791f1e29
paper_authors.name: Chongqing Zhao
paper_authors._id: 67298ecccdd83d27791f1e2a
paper_authors.name: Bo Wang
paper_authors._id: 67298ecccdd83d27791f1e2b
paper_authors.name: Hao Gong
paper_authors._id: 67298ecccdd83d27791f1e2c
paper_authors.name: Rong Gan
paper_authors._id: 67298ecccdd83d27791f1e2d
paper_authors.name: Winston Hu
paper_authors._id: 67298ecccdd83d27791f1e2e
paper_authors.name: Zhanhui Kang
paper_authors._id: 67298ecccdd83d27791f1e2f
paper_authors.name: Yong Yang
paper_authors._id: 67298ecccdd83d27791f1e30
paper_authors.name: Yuhong Liu
paper_authors._id: 67298ecccdd83d27791f1e31
paper_authors.name: Di Wang
paper_authors._id: 67298ecccdd83d27791f1e32
paper_authors.name: Jie Jiang
paper_title: Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
    Parameters by Tencent
paper_summary: In this paper, we introduce Hunyuan-Large, which is currently the largest
  open-source Transformer-based mixture of experts model, with a total of 389
  billion parameters and 52 billion activation parameters, capable of handling up
  to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior
  performance across various benchmarks including language understanding and
  generation, logical reasoning, mathematical problem-solving, coding,
  long-context, and aggregated tasks, where it outperforms LLama3.1-70B and
  exhibits comparable performance when compared to the significantly larger
  LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale
  synthetic data that is orders larger than in previous literature, a mixed
  expert routing strategy, a key-value cache compression technique, and an
  expert-specific learning rate strategy. Additionally, we also investigate the
  scaling laws and learning rate schedule of mixture of experts models, providing
  valuable insights and guidances for future model development and optimization.
  The code and checkpoints of Hunyuan-Large are released to facilitate future
  innovations and applications.
    Codes: https://github.com/Tencent/Hunyuan-Large
    Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large
paper_ai_keywords: Transformer-based mixture of experts parameter-efficient fine-tuning synthetic data mixed expert routing key-value cache compression expert-specific learning rate scaling laws learning rate schedule