Stacking Your Transformers: A Closer Look at Model Growth for Efficient
  LLM Pre-Training
LLMs are computationally expensive to pre-train due to their large scale.
Model growth emerges as a promising approach by leveraging smaller models to
accelerate the training of larger ones. However, the viability of these model
growth methods in efficient LLM pre-training remains underexplored. This work
identifies three critical textit{O}bstacles: (O1)
lack of comprehensive evaluation, (O2) untested viability for
scaling, and (O3) lack of empirical guidelines. To tackle
O1, we summarize existing approaches into four atomic growth
operators and systematically evaluate them in a standardized LLM pre-training
setting. Our findings reveal that a depthwise stacking operator, called
G_{stack}, exhibits remarkable acceleration in training, leading to
decreased loss and improved overall performance on eight standard NLP
benchmarks compared to strong baselines. Motivated by these promising results,
we conduct extensive experiments to delve deeper into G_{stack} to
address O2 and O3. For O2 (untested
scalability), our study shows that G_{stack} is scalable and
consistently performs well, with experiments up to 7B LLMs after growth and
pre-training LLMs with 750B tokens. For example, compared to a conventionally
trained 7B model using 300B tokens, our G_{stack} model converges to
the same loss with 194B tokens, resulting in a 54.6\% speedup. We further
address O3 (lack of empirical guidelines) by formalizing guidelines
to determine growth timing and growth factor for G_{stack}, making it
practical in general LLM pre-training. We also provide in-depth discussions and
comprehensive ablation studies of G_{stack}. Our code and pre-trained
model are available at
https://llm-stacking.github.io/{https://llm-stacking.github.io/}.
665401eff8137bb650bc1a60
Wenyu Du
claimed_verified
2024-06-21T07:07:27.519Z
624c3d2ca19f20b197761ba9
/avatars/7a64b81c29f4f6700fa18effc5616865.svg
Wenyu Du
user
wydu
665401eff8137bb650bc1a61
Tongxu Luo
claimed_verified
2024-05-27T08:49:39.691Z
6421b07e918f0fd889f0a682
/avatars/314b55c2428426c846d9449f98db4355.svg
Tongxu Luo
user
Zeno-Luo
665401eff8137bb650bc1a62
Zihan Qiu
665401eff8137bb650bc1a63
Zeyu Huang
claimed_verified
2024-08-14T12:42:34.994Z
64a5520dea3c5861303064b0
/avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
Zeyu hUANG
user
zeroyhuang
665401eff8137bb650bc1a64
Yikang Shen
claimed_verified
2024-06-25T20:04:14.029Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
665401eff8137bb650bc1a65
Reynold Cheng
665401eff8137bb650bc1a66
Yike Guo
665401eff8137bb650bc1a67
Jie Fu
claimed_verified
2024-05-27T07:17:51.338Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
Stacking Your Transformers: A Closer Look at Model Growth for Efficient
  LLM Pre-Training
LLMs are computationally expensive to pre-train due to their large scale.
Model growth emerges as a promising approach by leveraging smaller models to
accelerate the training of larger ones. However, the viability of these model
growth methods in efficient LLM pre-training remains underexplored. This work
identifies three critical textit{O}bstacles: (O1)
lack of comprehensive evaluation, (O2) untested viability for
scaling, and (O3) lack of empirical guidelines. To tackle
O1, we summarize existing approaches into four atomic growth
operators and systematically evaluate them in a standardized LLM pre-training
setting. Our findings reveal that a depthwise stacking operator, called
G_{stack}, exhibits remarkable acceleration in training, leading to
decreased loss and improved overall performance on eight standard NLP
benchmarks compared to strong baselines. Motivated by these promising results,
we conduct extensive experiments to delve deeper into G_{stack} to
address O2 and O3. For O2 (untested
scalability), our study shows that G_{stack} is scalable and
consistently performs well, with experiments up to 7B LLMs after growth and
pre-training LLMs with 750B tokens. For example, compared to a conventionally
trained 7B model using 300B tokens, our G_{stack} model converges to
the same loss with 194B tokens, resulting in a 54.6\% speedup. We further
address O3 (lack of empirical guidelines) by formalizing guidelines
to determine growth timing and growth factor for G_{stack}, making it
practical in general LLM pre-training. We also provide in-depth discussions and
comprehensive ablation studies of G_{stack}. Our code and pre-trained
model are available at
https://llm-stacking.github.io/{https://llm-stacking.github.io/}.
A depthwise stacking operator called $G_{\text{stack}}$ accelerates LLM pre-training, improving performance with less computational expense and providing guidelines for effective use.
LLMs pre-training model growth depthwise stacking operator $G_{\text{stack}}$ NLP benchmarks training acceleration loss reduction performance improvement scalability growth timing growth factor ablation studies