Rethinking Optimization and Architecture for Tiny Language Models
The power of large language models (LLMs) has been demonstrated through
numerous data and computing resources. However, the application of language
models on mobile devices is facing huge challenge on the computation and memory
costs, that is, tiny language models with high performance are urgently
required. Limited by the highly complex training process, there are many
details for optimizing language models that are seldom studied carefully. In
this study, based on a tiny language model with 1B parameters, we carefully
design a series of empirical study to analyze the effect of each component.
Three perspectives are mainly discussed, i.e., neural architecture, parameter
initialization, and optimization strategy. Several design formulas are
empirically proved especially effective for tiny language models, including
tokenizer compression, architecture tweaking, parameter inheritance and
multiple-round training. Then we train PanGu-pi-1B Pro and PanGu-pi-1.5B
Pro on 1.6T multilingual corpora, following the established formulas.
Experimental results demonstrate the improved optimization and architecture
yield a notable average improvement of 8.87 on benchmark evaluation sets for
PanGu-pi-1B Pro. Besides, PanGu-pi-1.5B Pro surpasses a range of SOTA
models with larger model sizes, validating its superior performance. The code
will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).
65c1b947d2bf7a4d69773cd9
Yehui Tang
65c1b947d2bf7a4d69773cda
Fangcheng Liu
65c1b947d2bf7a4d69773cdb
Yunsheng Ni
65c1b947d2bf7a4d69773cdc
Yuchuan Tian
65c1b947d2bf7a4d69773cdd
Zheyuan Bai
65c1b947d2bf7a4d69773cde
Yi-Qi Hu
65c1b947d2bf7a4d69773cdf
Sichao Liu
65c1b947d2bf7a4d69773ce0
Shangling Jui
65c1b947d2bf7a4d69773ce1
Kai Han
claimed_verified
2024-03-07T09:16:14.115Z
65e52e7d27dc8aa470a640e3
/avatars/022a179d14de29b9ab9d96fcc85aa264.svg
hankai
user
hankaixyz
65c1b947d2bf7a4d69773ce2
Yunhe Wang
Rethinking Optimization and Architecture for Tiny Language Models
The power of large language models (LLMs) has been demonstrated through
numerous data and computing resources. However, the application of language
models on mobile devices is facing huge challenge on the computation and memory
costs, that is, tiny language models with high performance are urgently
required. Limited by the highly complex training process, there are many
details for optimizing language models that are seldom studied carefully. In
this study, based on a tiny language model with 1B parameters, we carefully
design a series of empirical study to analyze the effect of each component.
Three perspectives are mainly discussed, i.e., neural architecture, parameter
initialization, and optimization strategy. Several design formulas are
empirically proved especially effective for tiny language models, including
tokenizer compression, architecture tweaking, parameter inheritance and
multiple-round training. Then we train PanGu-pi-1B Pro and PanGu-pi-1.5B
Pro on 1.6T multilingual corpora, following the established formulas.
Experimental results demonstrate the improved optimization and architecture
yield a notable average improvement of 8.87 on benchmark evaluation sets for
PanGu-pi-1B Pro. Besides, PanGu-pi-1.5B Pro surpasses a range of SOTA
models with larger model sizes, validating its superior performance. The code
will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).
Tiny language models optimized through tokenizer compression, architecture tweaking, and parameter inheritance achieve high performance on mobile devices with reduced computation and memory costs.
large language models tiny language models neural architecture parameter initialization optimization strategy tokenizer compression architecture tweaking parameter inheritance PanGu-$\pi$-1B Pro PanGu-$\pi$-1.5B Pro multilingual corpora benchmark evaluation sets