title: Rethinking Optimization and Architecture for Tiny Language Models
summary: The power of large language models (LLMs) has been demonstrated through
  numerous data and computing resources. However, the application of language
  models on mobile devices is facing huge challenge on the computation and memory
  costs, that is, tiny language models with high performance are urgently
  required. Limited by the highly complex training process, there are many
  details for optimizing language models that are seldom studied carefully. In
  this study, based on a tiny language model with 1B parameters, we carefully
  design a series of empirical study to analyze the effect of each component.
  Three perspectives are mainly discussed, i.e., neural architecture, parameter
  initialization, and optimization strategy. Several design formulas are
  empirically proved especially effective for tiny language models, including
  tokenizer compression, architecture tweaking, parameter inheritance and
  multiple-round training. Then we train PanGu-pi-1B Pro and PanGu-pi-1.5B
  Pro on 1.6T multilingual corpora, following the established formulas.
  Experimental results demonstrate the improved optimization and architecture
  yield a notable average improvement of 8.87 on benchmark evaluation sets for
  PanGu-pi-1B Pro. Besides, PanGu-pi-1.5B Pro surpasses a range of SOTA
  models with larger model sizes, validating its superior performance. The code
  will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).
paper_authors._id: 65c1b947d2bf7a4d69773cd9
paper_authors.name: Yehui Tang
paper_authors._id: 65c1b947d2bf7a4d69773cda
paper_authors.name: Fangcheng Liu
paper_authors._id: 65c1b947d2bf7a4d69773cdb
paper_authors.name: Yunsheng Ni
paper_authors._id: 65c1b947d2bf7a4d69773cdc
paper_authors.name: Yuchuan Tian
paper_authors._id: 65c1b947d2bf7a4d69773cdd
paper_authors.name: Zheyuan Bai
paper_authors._id: 65c1b947d2bf7a4d69773cde
paper_authors.name: Yi-Qi Hu
paper_authors._id: 65c1b947d2bf7a4d69773cdf
paper_authors.name: Sichao Liu
paper_authors._id: 65c1b947d2bf7a4d69773ce0
paper_authors.name: Shangling Jui
paper_authors._id: 65c1b947d2bf7a4d69773ce1
paper_authors.name: Kai Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:16:14.115Z
paper_authors._id: 65e52e7d27dc8aa470a640e3
paper_authors.avatarUrl: /avatars/022a179d14de29b9ab9d96fcc85aa264.svg
paper_authors.fullname: hankai
paper_authors.type: user
paper_authors.user: hankaixyz
paper_authors._id: 65c1b947d2bf7a4d69773ce2
paper_authors.name: Yunhe Wang
paper_title: Rethinking Optimization and Architecture for Tiny Language Models
paper_summary: The power of large language models (LLMs) has been demonstrated through
  numerous data and computing resources. However, the application of language
  models on mobile devices is facing huge challenge on the computation and memory
  costs, that is, tiny language models with high performance are urgently
  required. Limited by the highly complex training process, there are many
  details for optimizing language models that are seldom studied carefully. In
  this study, based on a tiny language model with 1B parameters, we carefully
  design a series of empirical study to analyze the effect of each component.
  Three perspectives are mainly discussed, i.e., neural architecture, parameter
  initialization, and optimization strategy. Several design formulas are
  empirically proved especially effective for tiny language models, including
  tokenizer compression, architecture tweaking, parameter inheritance and
  multiple-round training. Then we train PanGu-pi-1B Pro and PanGu-pi-1.5B
  Pro on 1.6T multilingual corpora, following the established formulas.
  Experimental results demonstrate the improved optimization and architecture
  yield a notable average improvement of 8.87 on benchmark evaluation sets for
  PanGu-pi-1B Pro. Besides, PanGu-pi-1.5B Pro surpasses a range of SOTA
  models with larger model sizes, validating its superior performance. The code
  will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).
paper_ai_summary: Tiny language models optimized through tokenizer compression, architecture tweaking, and parameter inheritance achieve high performance on mobile devices with reduced computation and memory costs.
paper_ai_keywords: large language models tiny language models neural architecture parameter initialization optimization strategy tokenizer compression architecture tweaking parameter inheritance PanGu-$\pi$-1B Pro PanGu-$\pi$-1.5B Pro multilingual corpora benchmark evaluation sets