title: PanGu-π: Enhancing Language Model Architectures via Nonlinearity
    Compensation
summary: The recent trend of large language models (LLMs) is to increase the scale of
  both model size (\aka the number of parameters) and dataset to achieve better
  generative ability, which is definitely proved by a lot of work such as the
  famous GPT and Llama. However, large models often involve massive computational
  costs, and practical applications cannot afford such high prices. However, the
  method of constructing a strong model architecture for LLMs is rarely
  discussed. We first analyze the state-of-the-art language model architectures
  and observe the feature collapse problem. Based on the theoretical analysis, we
  propose that the nonlinearity is also very important for language models, which
  is usually studied in convolutional neural networks for vision tasks. The
  series informed activation function is then introduced with tiny calculations
  that can be ignored, and an augmented shortcut is further used to enhance the
  model nonlinearity. We then demonstrate that the proposed approach is
  significantly effective for enhancing the model nonlinearity through carefully
  designed ablations; thus, we present a new efficient model architecture for
  establishing modern, namely, PanGu-pi. Experiments are then conducted using
  the same dataset and training strategy to compare PanGu-pi with
  state-of-the-art LLMs. The results show that PanGu-pi-7B can achieve a
  comparable performance to that of benchmarks with about 10\% inference
  speed-up, and PanGu-pi-1B can achieve state-of-the-art performance in terms
  of accuracy and efficiency. In addition, we have deployed PanGu-pi-7B in the
  high-value domains of finance and law, developing an LLM named YunShan for
  practical application. The results show that YunShan can surpass other models
  with similar scales on benchmarks.
paper_authors._id: 65922e1e7fe0235473779d2d
paper_authors.name: Yunhe Wang
paper_authors._id: 65922e1e7fe0235473779d2e
paper_authors.name: Hanting Chen
paper_authors._id: 65922e1e7fe0235473779d2f
paper_authors.name: Yehui Tang
paper_authors._id: 65922e1e7fe0235473779d30
paper_authors.name: Tianyu Guo
paper_authors._id: 65922e1e7fe0235473779d31
paper_authors.name: Kai Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:16:55.381Z
paper_authors._id: 65e52e7d27dc8aa470a640e3
paper_authors.avatarUrl: /avatars/022a179d14de29b9ab9d96fcc85aa264.svg
paper_authors.fullname: hankai
paper_authors.type: user
paper_authors.user: hankaixyz
paper_authors._id: 65922e1e7fe0235473779d32
paper_authors.name: Ying Nie
paper_authors._id: 65922e1e7fe0235473779d33
paper_authors.name: Xutao Wang
paper_authors._id: 65922e1e7fe0235473779d34
paper_authors.name: Hailin Hu
paper_authors._id: 65922e1e7fe0235473779d35
paper_authors.name: Zheyuan Bai
paper_authors._id: 65922e1e7fe0235473779d36
paper_authors.name: Yun Wang
paper_authors._id: 65922e1e7fe0235473779d37
paper_authors.name: Fangcheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:26:47.953Z
paper_authors._id: 64b78295479b934973e2c40e
paper_authors.avatarUrl: /avatars/9213e385964132fa50859264a838d891.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: Fangcheng2
paper_authors._id: 65922e1e7fe0235473779d38
paper_authors.name: Zhicheng Liu
paper_authors._id: 65922e1e7fe0235473779d39
paper_authors.name: Jianyuan Guo
paper_authors._id: 65922e1e7fe0235473779d3a
paper_authors.name: Sinan Zeng
paper_authors._id: 65922e1e7fe0235473779d3b
paper_authors.name: Yinchen Zhang
paper_authors._id: 65922e1e7fe0235473779d3c
paper_authors.name: Qinghua Xu
paper_authors._id: 65922e1e7fe0235473779d3d
paper_authors.name: Qun Liu
paper_authors._id: 65922e1e7fe0235473779d3e
paper_authors.name: Jun Yao
paper_authors._id: 65922e1e7fe0235473779d3f
paper_authors.name: Chao Xu
paper_authors._id: 65922e1e7fe0235473779d40
paper_authors.name: Dacheng Tao
paper_title: PanGu-π: Enhancing Language Model Architectures via Nonlinearity
    Compensation
paper_summary: The recent trend of large language models (LLMs) is to increase the scale of
  both model size (\aka the number of parameters) and dataset to achieve better
  generative ability, which is definitely proved by a lot of work such as the
  famous GPT and Llama. However, large models often involve massive computational
  costs, and practical applications cannot afford such high prices. However, the
  method of constructing a strong model architecture for LLMs is rarely
  discussed. We first analyze the state-of-the-art language model architectures
  and observe the feature collapse problem. Based on the theoretical analysis, we
  propose that the nonlinearity is also very important for language models, which
  is usually studied in convolutional neural networks for vision tasks. The
  series informed activation function is then introduced with tiny calculations
  that can be ignored, and an augmented shortcut is further used to enhance the
  model nonlinearity. We then demonstrate that the proposed approach is
  significantly effective for enhancing the model nonlinearity through carefully
  designed ablations; thus, we present a new efficient model architecture for
  establishing modern, namely, PanGu-pi. Experiments are then conducted using
  the same dataset and training strategy to compare PanGu-pi with
  state-of-the-art LLMs. The results show that PanGu-pi-7B can achieve a
  comparable performance to that of benchmarks with about 10\% inference
  speed-up, and PanGu-pi-1B can achieve state-of-the-art performance in terms
  of accuracy and efficiency. In addition, we have deployed PanGu-pi-7B in the
  high-value domains of finance and law, developing an LLM named YunShan for
  practical application. The results show that YunShan can surpass other models
  with similar scales on benchmarks.
paper_ai_summary: The paper proposes a new efficient architecture for large language models by incorporating informed activation functions and augmented shortcuts to enhance nonlinearity, resulting in improved performance and speed compared to state-of-the-art models.
paper_ai_keywords: large language models LLMs feature collapse nonlinearity convolutional neural networks informed activation function augmented shortcut PanGu-π YunShan