title: LLaMA Pro: Progressive LLaMA with Block Expansion
summary: Humans generally acquire new skills without compromising the old; however,
  the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to
  CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with
  an expansion of Transformer blocks. We tune the expanded blocks using only new
  corpus, efficiently and effectively improving the model's knowledge without
  catastrophic forgetting. In this paper, we experiment on the corpus of code and
  math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from
  LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro
  and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced
  performance among various benchmarks, demonstrating superiority over existing
  open models in the LLaMA family and the immense potential of reasoning and
  addressing diverse tasks as an intelligent agent. Our findings provide valuable
  insights into integrating natural and programming languages, laying a solid
  foundation for developing advanced language agents that operate effectively in
  various environments.
paper_authors._id: 65978d26b4b5c254cb8b0924
paper_authors.name: Chengyue Wu
paper_authors._id: 65978d26b4b5c254cb8b0925
paper_authors.name: Yukang Gan
paper_authors._id: 65978d26b4b5c254cb8b0926
paper_authors.name: Yixiao Ge
paper_authors._id: 65978d26b4b5c254cb8b0927
paper_authors.name: Zeyu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T12:00:53.981Z
paper_authors._id: 635626a8ec32331b227f407b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635626a8ec32331b227f407b/KRkAEN3eXN_mYzzJQ_8dO.jpeg
paper_authors.fullname: LuZeyu
paper_authors.type: user
paper_authors.user: whlzy
paper_authors._id: 65978d26b4b5c254cb8b0928
paper_authors.name: Jiahao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T09:02:17.717Z
paper_authors._id: 65994517d3f2137415bb4d5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
paper_authors.fullname: WANG Jiahao
paper_authors.type: user
paper_authors.user: techmonsterwang
paper_authors._id: 65978d26b4b5c254cb8b0929
paper_authors.name: Ye Feng
paper_authors._id: 65978d26b4b5c254cb8b092a
paper_authors.name: Ping Luo
paper_authors._id: 65978d26b4b5c254cb8b092b
paper_authors.name: Ying Shan
paper_title: LLaMA Pro: Progressive LLaMA with Block Expansion
paper_summary: Humans generally acquire new skills without compromising the old; however,
  the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to
  CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with
  an expansion of Transformer blocks. We tune the expanded blocks using only new
  corpus, efficiently and effectively improving the model's knowledge without
  catastrophic forgetting. In this paper, we experiment on the corpus of code and
  math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from
  LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro
  and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced
  performance among various benchmarks, demonstrating superiority over existing
  open models in the LLaMA family and the immense potential of reasoning and
  addressing diverse tasks as an intelligent agent. Our findings provide valuable
  insights into integrating natural and programming languages, laying a solid
  foundation for developing advanced language agents that operate effectively in
  various environments.
paper_ai_keywords: Transformer blocks post-pretraining catastrophic forgetting LLaMA LLaMA Pro-8.3B LLaMA2-7B instruction-following benchmarks reasoning language agents