Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend
  NPUs
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion
parameters and dense Transformer modules trained on Ascend Neural Processing
Units (NPUs). Although the field of LLM has been witnessing unprecedented
advances in pushing the scale and capability of LLM in recent years, training
such a large-scale model still involves significant optimization and system
challenges. To stabilize the training process, we propose depth-scaled sandwich
normalization, which effectively eliminates loss spikes during the training
process of deep models. We pre-train our model on 13.2 trillion diverse and
high-quality tokens and further enhance its reasoning capabilities during
post-training. To perform such large-scale training efficiently, we utilize
8,192 Ascend NPUs with a series of system optimizations. Evaluations on
multiple diverse benchmarks indicate that Pangu Ultra significantly advances
the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral
Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse
model structure contains much more parameters. Our exploration demonstrates
that Ascend NPUs are capable of efficiently and effectively training dense
models with more than 100 billion parameters. Our model and system will be
available for our commercial customers.
67f90407614ff5204ab48c23
Yichun Yin
67f90407614ff5204ab48c24
Wenyong Huang
67f90407614ff5204ab48c25
Kaikai Song
67f90407614ff5204ab48c26
Yehui Tang
67f90407614ff5204ab48c27
Xueyu Wu
67f90407614ff5204ab48c28
Wei Guo
67f90407614ff5204ab48c29
Peng Guo
67f90407614ff5204ab48c2a
Yaoyuan Wang
67f90407614ff5204ab48c2b
Xiaojun Meng
67f90407614ff5204ab48c2c
Yasheng Wang
67f90407614ff5204ab48c2d
Dong Li
67f90407614ff5204ab48c2e
Can Chen
67f90407614ff5204ab48c2f
Dandan Tu
67f90407614ff5204ab48c30
Yin Li
67f90407614ff5204ab48c31
Fisher Yu
67f90407614ff5204ab48c32
Ruiming Tang
67f90407614ff5204ab48c33
Yunhe Wang
67f90407614ff5204ab48c34
Baojun Wang
67f90407614ff5204ab48c35
Bin Wang
67f90407614ff5204ab48c36
Bo Wang
67f90407614ff5204ab48c37
Boxiao Liu
67f90407614ff5204ab48c38
Changzheng Zhang
67f90407614ff5204ab48c39
Duyu Tang
67f90407614ff5204ab48c3a
Fei Mi
67f90407614ff5204ab48c3b
Hui Jin
67f90407614ff5204ab48c3c
Jiansheng Wei
67f90407614ff5204ab48c3d
Jiarui Qin
67f90407614ff5204ab48c3e
Jinpeng Li
67f90407614ff5204ab48c3f
Jun Zhao
67f90407614ff5204ab48c40
Liqun Deng
67f90407614ff5204ab48c41
Lin Li
67f90407614ff5204ab48c42
Minghui Xu
67f90407614ff5204ab48c43
Naifu Zhang
67f90407614ff5204ab48c44
Nianzu Zheng
67f90407614ff5204ab48c45
Qiang Li
67f90407614ff5204ab48c46
Rongju Ruan
67f90407614ff5204ab48c47
Shengjun Cheng
67f90407614ff5204ab48c48
Tianyu Guo
67f90407614ff5204ab48c49
Wei He
67f90407614ff5204ab48c4a
Wei Li
67f90407614ff5204ab48c4b
Weiwen Liu
67f90407614ff5204ab48c4c
Wulong Liu
67f90407614ff5204ab48c4d
Xinyi Dai
67f90407614ff5204ab48c4e
Yonghan Dong
67f90407614ff5204ab48c4f
Yu Pan
67f90407614ff5204ab48c50
Yue Li
67f90407614ff5204ab48c51
Yufei Wang
67f90407614ff5204ab48c52
Yujun Li
67f90407614ff5204ab48c53
Yunsheng Ni
67f90407614ff5204ab48c54
Zhe Liu
67f90407614ff5204ab48c55
Zhenhe Zhang
67f90407614ff5204ab48c56
Zhicheng Liu
Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend
  NPUs
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion
parameters and dense Transformer modules trained on Ascend Neural Processing
Units (NPUs). Although the field of LLM has been witnessing unprecedented
advances in pushing the scale and capability of LLM in recent years, training
such a large-scale model still involves significant optimization and system
challenges. To stabilize the training process, we propose depth-scaled sandwich
normalization, which effectively eliminates loss spikes during the training
process of deep models. We pre-train our model on 13.2 trillion diverse and
high-quality tokens and further enhance its reasoning capabilities during
post-training. To perform such large-scale training efficiently, we utilize
8,192 Ascend NPUs with a series of system optimizations. Evaluations on
multiple diverse benchmarks indicate that Pangu Ultra significantly advances
the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral
Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse
model structure contains much more parameters. Our exploration demonstrates
that Ascend NPUs are capable of efficiently and effectively training dense
models with more than 100 billion parameters. Our model and system will be
available for our commercial customers.
Pangu Ultra, a large language model with 135 billion parameters, achieves state-of-the-art performance using depth-scaled sandwich normalization and efficient training on Ascend NPUs.
Large Language Model (LLM) dense Transformer modules Ascend Neural Processing Units (NPUs) depth-scaled sandwich normalization pre-training post-training Llama 405B Mistral Large 2 DeepSeek-R1 sparse model structure