PanGu-π: Enhancing Language Model Architectures via Nonlinearity
  Compensation
The recent trend of large language models (LLMs) is to increase the scale of
both model size (\aka the number of parameters) and dataset to achieve better
generative ability, which is definitely proved by a lot of work such as the
famous GPT and Llama. However, large models often involve massive computational
costs, and practical applications cannot afford such high prices. However, the
method of constructing a strong model architecture for LLMs is rarely
discussed. We first analyze the state-of-the-art language model architectures
and observe the feature collapse problem. Based on the theoretical analysis, we
propose that the nonlinearity is also very important for language models, which
is usually studied in convolutional neural networks for vision tasks. The
series informed activation function is then introduced with tiny calculations
that can be ignored, and an augmented shortcut is further used to enhance the
model nonlinearity. We then demonstrate that the proposed approach is
significantly effective for enhancing the model nonlinearity through carefully
designed ablations; thus, we present a new efficient model architecture for
establishing modern, namely, PanGu-pi. Experiments are then conducted using
the same dataset and training strategy to compare PanGu-pi with
state-of-the-art LLMs. The results show that PanGu-pi-7B can achieve a
comparable performance to that of benchmarks with about 10\% inference
speed-up, and PanGu-pi-1B can achieve state-of-the-art performance in terms
of accuracy and efficiency. In addition, we have deployed PanGu-pi-7B in the
high-value domains of finance and law, developing an LLM named YunShan for
practical application. The results show that YunShan can surpass other models
with similar scales on benchmarks.
65922e1e7fe0235473779d2d
Yunhe Wang
65922e1e7fe0235473779d2e
Hanting Chen
65922e1e7fe0235473779d2f
Yehui Tang
65922e1e7fe0235473779d30
Tianyu Guo
65922e1e7fe0235473779d31
Kai Han
claimed_verified
2024-03-07T09:16:55.381Z
65e52e7d27dc8aa470a640e3
/avatars/022a179d14de29b9ab9d96fcc85aa264.svg
hankai
user
hankaixyz
65922e1e7fe0235473779d32
Ying Nie
65922e1e7fe0235473779d33
Xutao Wang
65922e1e7fe0235473779d34
Hailin Hu
65922e1e7fe0235473779d35
Zheyuan Bai
65922e1e7fe0235473779d36
Yun Wang
65922e1e7fe0235473779d37
Fangcheng Liu
claimed_verified
2024-01-29T08:26:47.953Z
64b78295479b934973e2c40e
/avatars/9213e385964132fa50859264a838d891.svg
liu
user
Fangcheng2
65922e1e7fe0235473779d38
Zhicheng Liu
65922e1e7fe0235473779d39
Jianyuan Guo
65922e1e7fe0235473779d3a
Sinan Zeng
65922e1e7fe0235473779d3b
Yinchen Zhang
65922e1e7fe0235473779d3c
Qinghua Xu
65922e1e7fe0235473779d3d
Qun Liu
65922e1e7fe0235473779d3e
Jun Yao
65922e1e7fe0235473779d3f
Chao Xu
65922e1e7fe0235473779d40
Dacheng Tao
PanGu-π: Enhancing Language Model Architectures via Nonlinearity
  Compensation
The recent trend of large language models (LLMs) is to increase the scale of
both model size (\aka the number of parameters) and dataset to achieve better
generative ability, which is definitely proved by a lot of work such as the
famous GPT and Llama. However, large models often involve massive computational
costs, and practical applications cannot afford such high prices. However, the
method of constructing a strong model architecture for LLMs is rarely
discussed. We first analyze the state-of-the-art language model architectures
and observe the feature collapse problem. Based on the theoretical analysis, we
propose that the nonlinearity is also very important for language models, which
is usually studied in convolutional neural networks for vision tasks. The
series informed activation function is then introduced with tiny calculations
that can be ignored, and an augmented shortcut is further used to enhance the
model nonlinearity. We then demonstrate that the proposed approach is
significantly effective for enhancing the model nonlinearity through carefully
designed ablations; thus, we present a new efficient model architecture for
establishing modern, namely, PanGu-pi. Experiments are then conducted using
the same dataset and training strategy to compare PanGu-pi with
state-of-the-art LLMs. The results show that PanGu-pi-7B can achieve a
comparable performance to that of benchmarks with about 10\% inference
speed-up, and PanGu-pi-1B can achieve state-of-the-art performance in terms
of accuracy and efficiency. In addition, we have deployed PanGu-pi-7B in the
high-value domains of finance and law, developing an LLM named YunShan for
practical application. The results show that YunShan can surpass other models
with similar scales on benchmarks.
The paper proposes a new efficient architecture for large language models by incorporating informed activation functions and augmented shortcuts to enhance nonlinearity, resulting in improved performance and speed compared to state-of-the-art models.
large language models LLMs feature collapse nonlinearity convolutional neural networks informed activation function augmented shortcut PanGu-π YunShan