LLaMA Pro: Progressive LLaMA with Block Expansion
Humans generally acquire new skills without compromising the old; however,
the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to
CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with
an expansion of Transformer blocks. We tune the expanded blocks using only new
corpus, efficiently and effectively improving the model's knowledge without
catastrophic forgetting. In this paper, we experiment on the corpus of code and
math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from
LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro
and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced
performance among various benchmarks, demonstrating superiority over existing
open models in the LLaMA family and the immense potential of reasoning and
addressing diverse tasks as an intelligent agent. Our findings provide valuable
insights into integrating natural and programming languages, laying a solid
foundation for developing advanced language agents that operate effectively in
various environments.
65978d26b4b5c254cb8b0924
Chengyue Wu
65978d26b4b5c254cb8b0925
Yukang Gan
65978d26b4b5c254cb8b0926
Yixiao Ge
65978d26b4b5c254cb8b0927
Zeyu Lu
claimed_verified
2024-09-04T12:00:53.981Z
635626a8ec32331b227f407b
https://cdn-avatars.huggingface.co/v1/production/uploads/635626a8ec32331b227f407b/KRkAEN3eXN_mYzzJQ_8dO.jpeg
LuZeyu
user
whlzy
65978d26b4b5c254cb8b0928
Jiahao Wang
claimed_verified
2024-04-11T09:02:17.717Z
65994517d3f2137415bb4d5b
https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
WANG Jiahao
user
techmonsterwang
65978d26b4b5c254cb8b0929
Ye Feng
65978d26b4b5c254cb8b092a
Ping Luo
65978d26b4b5c254cb8b092b
Ying Shan
LLaMA Pro: Progressive LLaMA with Block Expansion
Humans generally acquire new skills without compromising the old; however,
the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to
CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with
an expansion of Transformer blocks. We tune the expanded blocks using only new
corpus, efficiently and effectively improving the model's knowledge without
catastrophic forgetting. In this paper, we experiment on the corpus of code and
math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from
LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro
and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced
performance among various benchmarks, demonstrating superiority over existing
open models in the LLaMA family and the immense potential of reasoning and
addressing diverse tasks as an intelligent agent. Our findings provide valuable
insights into integrating natural and programming languages, laying a solid
foundation for developing advanced language agents that operate effectively in
various environments.
A new post-pretraining method using expanded Transformer blocks for Large Language Models improves knowledge without catastrophic forgetting, yielding LLaMA Pro-8.3B that excels in general tasks, programming, and mathematics.
Large Language Models LLMs post-pretraining method Transformer blocks catastrophic forgetting LLaMA CodeLLaMA LLaMA Pro-8.3B instruction-following advanced performance benchmarks reasoning intelligent agent