title: Stacking Your Transformers: A Closer Look at Model Growth for Efficient
    LLM Pre-Training
summary: LLMs are computationally expensive to pre-train due to their large scale.
  Model growth emerges as a promising approach by leveraging smaller models to
  accelerate the training of larger ones. However, the viability of these model
  growth methods in efficient LLM pre-training remains underexplored. This work
  identifies three critical textit{O}bstacles: (O1)
  lack of comprehensive evaluation, (O2) untested viability for
  scaling, and (O3) lack of empirical guidelines. To tackle
  O1, we summarize existing approaches into four atomic growth
  operators and systematically evaluate them in a standardized LLM pre-training
  setting. Our findings reveal that a depthwise stacking operator, called
  G_{stack}, exhibits remarkable acceleration in training, leading to
  decreased loss and improved overall performance on eight standard NLP
  benchmarks compared to strong baselines. Motivated by these promising results,
  we conduct extensive experiments to delve deeper into G_{stack} to
  address O2 and O3. For O2 (untested
  scalability), our study shows that G_{stack} is scalable and
  consistently performs well, with experiments up to 7B LLMs after growth and
  pre-training LLMs with 750B tokens. For example, compared to a conventionally
  trained 7B model using 300B tokens, our G_{stack} model converges to
  the same loss with 194B tokens, resulting in a 54.6\% speedup. We further
  address O3 (lack of empirical guidelines) by formalizing guidelines
  to determine growth timing and growth factor for G_{stack}, making it
  practical in general LLM pre-training. We also provide in-depth discussions and
  comprehensive ablation studies of G_{stack}. Our code and pre-trained
  model are available at
  https://llm-stacking.github.io/{https://llm-stacking.github.io/}.
paper_authors._id: 665401eff8137bb650bc1a60
paper_authors.name: Wenyu Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:27.519Z
paper_authors._id: 624c3d2ca19f20b197761ba9
paper_authors.avatarUrl: /avatars/7a64b81c29f4f6700fa18effc5616865.svg
paper_authors.fullname: Wenyu Du
paper_authors.type: user
paper_authors.user: wydu
paper_authors._id: 665401eff8137bb650bc1a61
paper_authors.name: Tongxu Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T08:49:39.691Z
paper_authors._id: 6421b07e918f0fd889f0a682
paper_authors.avatarUrl: /avatars/314b55c2428426c846d9449f98db4355.svg
paper_authors.fullname: Tongxu Luo
paper_authors.type: user
paper_authors.user: Kuroxiro
paper_authors._id: 665401eff8137bb650bc1a62
paper_authors.name: Zihan Qiu
paper_authors._id: 665401eff8137bb650bc1a63
paper_authors.name: Zeyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T12:42:34.994Z
paper_authors._id: 64a5520dea3c5861303064b0
paper_authors.avatarUrl: /avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
paper_authors.fullname: Zeyu hUANG
paper_authors.type: user
paper_authors.user: zeroyhuang
paper_authors._id: 665401eff8137bb650bc1a64
paper_authors.name: Yikang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:04:14.029Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 665401eff8137bb650bc1a65
paper_authors.name: Reynold Cheng
paper_authors._id: 665401eff8137bb650bc1a66
paper_authors.name: Yike Guo
paper_authors._id: 665401eff8137bb650bc1a67
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:17:51.338Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: Stacking Your Transformers: A Closer Look at Model Growth for Efficient
    LLM Pre-Training
paper_summary: LLMs are computationally expensive to pre-train due to their large scale.
  Model growth emerges as a promising approach by leveraging smaller models to
  accelerate the training of larger ones. However, the viability of these model
  growth methods in efficient LLM pre-training remains underexplored. This work
  identifies three critical textit{O}bstacles: (O1)
  lack of comprehensive evaluation, (O2) untested viability for
  scaling, and (O3) lack of empirical guidelines. To tackle
  O1, we summarize existing approaches into four atomic growth
  operators and systematically evaluate them in a standardized LLM pre-training
  setting. Our findings reveal that a depthwise stacking operator, called
  G_{stack}, exhibits remarkable acceleration in training, leading to
  decreased loss and improved overall performance on eight standard NLP
  benchmarks compared to strong baselines. Motivated by these promising results,
  we conduct extensive experiments to delve deeper into G_{stack} to
  address O2 and O3. For O2 (untested
  scalability), our study shows that G_{stack} is scalable and
  consistently performs well, with experiments up to 7B LLMs after growth and
  pre-training LLMs with 750B tokens. For example, compared to a conventionally
  trained 7B model using 300B tokens, our G_{stack} model converges to
  the same loss with 194B tokens, resulting in a 54.6\% speedup. We further
  address O3 (lack of empirical guidelines) by formalizing guidelines
  to determine growth timing and growth factor for G_{stack}, making it
  practical in general LLM pre-training. We also provide in-depth discussions and
  comprehensive ablation studies of G_{stack}. Our code and pre-trained
  model are available at
  https://llm-stacking.github.io/{https://llm-stacking.github.io/}.
paper_ai_keywords: LLMs pre-training model growth depthwise stacking $G_{\text{stack}}$ NLP benchmarks LLM pre-training setting growth operators scalability growth timing growth factor ablation studies