title: Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend
    NPUs
summary: We present Pangu Ultra, a Large Language Model (LLM) with 135 billion
  parameters and dense Transformer modules trained on Ascend Neural Processing
  Units (NPUs). Although the field of LLM has been witnessing unprecedented
  advances in pushing the scale and capability of LLM in recent years, training
  such a large-scale model still involves significant optimization and system
  challenges. To stabilize the training process, we propose depth-scaled sandwich
  normalization, which effectively eliminates loss spikes during the training
  process of deep models. We pre-train our model on 13.2 trillion diverse and
  high-quality tokens and further enhance its reasoning capabilities during
  post-training. To perform such large-scale training efficiently, we utilize
  8,192 Ascend NPUs with a series of system optimizations. Evaluations on
  multiple diverse benchmarks indicate that Pangu Ultra significantly advances
  the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral
  Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse
  model structure contains much more parameters. Our exploration demonstrates
  that Ascend NPUs are capable of efficiently and effectively training dense
  models with more than 100 billion parameters. Our model and system will be
  available for our commercial customers.
paper_authors._id: 67f90407614ff5204ab48c23
paper_authors.name: Yichun Yin
paper_authors._id: 67f90407614ff5204ab48c24
paper_authors.name: Wenyong Huang
paper_authors._id: 67f90407614ff5204ab48c25
paper_authors.name: Kaikai Song
paper_authors._id: 67f90407614ff5204ab48c26
paper_authors.name: Yehui Tang
paper_authors._id: 67f90407614ff5204ab48c27
paper_authors.name: Xueyu Wu
paper_authors._id: 67f90407614ff5204ab48c28
paper_authors.name: Wei Guo
paper_authors._id: 67f90407614ff5204ab48c29
paper_authors.name: Peng Guo
paper_authors._id: 67f90407614ff5204ab48c2a
paper_authors.name: Yaoyuan Wang
paper_authors._id: 67f90407614ff5204ab48c2b
paper_authors.name: Xiaojun Meng
paper_authors._id: 67f90407614ff5204ab48c2c
paper_authors.name: Yasheng Wang
paper_authors._id: 67f90407614ff5204ab48c2d
paper_authors.name: Dong Li
paper_authors._id: 67f90407614ff5204ab48c2e
paper_authors.name: Can Chen
paper_authors._id: 67f90407614ff5204ab48c2f
paper_authors.name: Dandan Tu
paper_authors._id: 67f90407614ff5204ab48c30
paper_authors.name: Yin Li
paper_authors._id: 67f90407614ff5204ab48c31
paper_authors.name: Fisher Yu
paper_authors._id: 67f90407614ff5204ab48c32
paper_authors.name: Ruiming Tang
paper_authors._id: 67f90407614ff5204ab48c33
paper_authors.name: Yunhe Wang
paper_authors._id: 67f90407614ff5204ab48c34
paper_authors.name: Baojun Wang
paper_authors._id: 67f90407614ff5204ab48c35
paper_authors.name: Bin Wang
paper_authors._id: 67f90407614ff5204ab48c36
paper_authors.name: Bo Wang
paper_authors._id: 67f90407614ff5204ab48c37
paper_authors.name: Boxiao Liu
paper_authors._id: 67f90407614ff5204ab48c38
paper_authors.name: Changzheng Zhang
paper_authors._id: 67f90407614ff5204ab48c39
paper_authors.name: Duyu Tang
paper_authors._id: 67f90407614ff5204ab48c3a
paper_authors.name: Fei Mi
paper_authors._id: 67f90407614ff5204ab48c3b
paper_authors.name: Hui Jin
paper_authors._id: 67f90407614ff5204ab48c3c
paper_authors.name: Jiansheng Wei
paper_authors._id: 67f90407614ff5204ab48c3d
paper_authors.name: Jiarui Qin
paper_authors._id: 67f90407614ff5204ab48c3e
paper_authors.name: Jinpeng Li
paper_authors._id: 67f90407614ff5204ab48c3f
paper_authors.name: Jun Zhao
paper_authors._id: 67f90407614ff5204ab48c40
paper_authors.name: Liqun Deng
paper_authors._id: 67f90407614ff5204ab48c41
paper_authors.name: Lin Li
paper_authors._id: 67f90407614ff5204ab48c42
paper_authors.name: Minghui Xu
paper_authors._id: 67f90407614ff5204ab48c43
paper_authors.name: Naifu Zhang
paper_authors._id: 67f90407614ff5204ab48c44
paper_authors.name: Nianzu Zheng
paper_authors._id: 67f90407614ff5204ab48c45
paper_authors.name: Qiang Li
paper_authors._id: 67f90407614ff5204ab48c46
paper_authors.name: Rongju Ruan
paper_authors._id: 67f90407614ff5204ab48c47
paper_authors.name: Shengjun Cheng
paper_authors._id: 67f90407614ff5204ab48c48
paper_authors.name: Tianyu Guo
paper_authors._id: 67f90407614ff5204ab48c49
paper_authors.name: Wei He
paper_authors._id: 67f90407614ff5204ab48c4a
paper_authors.name: Wei Li
paper_authors._id: 67f90407614ff5204ab48c4b
paper_authors.name: Weiwen Liu
paper_authors._id: 67f90407614ff5204ab48c4c
paper_authors.name: Wulong Liu
paper_authors._id: 67f90407614ff5204ab48c4d
paper_authors.name: Xinyi Dai
paper_authors._id: 67f90407614ff5204ab48c4e
paper_authors.name: Yonghan Dong
paper_authors._id: 67f90407614ff5204ab48c4f
paper_authors.name: Yu Pan
paper_authors._id: 67f90407614ff5204ab48c50
paper_authors.name: Yue Li
paper_authors._id: 67f90407614ff5204ab48c51
paper_authors.name: Yufei Wang
paper_authors._id: 67f90407614ff5204ab48c52
paper_authors.name: Yujun Li
paper_authors._id: 67f90407614ff5204ab48c53
paper_authors.name: Yunsheng Ni
paper_authors._id: 67f90407614ff5204ab48c54
paper_authors.name: Zhe Liu
paper_authors._id: 67f90407614ff5204ab48c55
paper_authors.name: Zhenhe Zhang
paper_authors._id: 67f90407614ff5204ab48c56
paper_authors.name: Zhicheng Liu
paper_title: Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend
    NPUs
paper_summary: We present Pangu Ultra, a Large Language Model (LLM) with 135 billion
  parameters and dense Transformer modules trained on Ascend Neural Processing
  Units (NPUs). Although the field of LLM has been witnessing unprecedented
  advances in pushing the scale and capability of LLM in recent years, training
  such a large-scale model still involves significant optimization and system
  challenges. To stabilize the training process, we propose depth-scaled sandwich
  normalization, which effectively eliminates loss spikes during the training
  process of deep models. We pre-train our model on 13.2 trillion diverse and
  high-quality tokens and further enhance its reasoning capabilities during
  post-training. To perform such large-scale training efficiently, we utilize
  8,192 Ascend NPUs with a series of system optimizations. Evaluations on
  multiple diverse benchmarks indicate that Pangu Ultra significantly advances
  the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral
  Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse
  model structure contains much more parameters. Our exploration demonstrates
  that Ascend NPUs are capable of efficiently and effectively training dense
  models with more than 100 billion parameters. Our model and system will be
  available for our commercial customers.
paper_ai_summary: Pangu Ultra, a large language model with 135 billion parameters, achieves state-of-the-art performance using depth-scaled sandwich normalization and efficient training on Ascend NPUs.
paper_ai_keywords: Large Language Model (LLM) dense Transformer modules Ascend Neural Processing Units (NPUs) depth-scaled sandwich normalization pre-training post-training Llama 405B Mistral Large 2 DeepSeek-R1 sparse model structure