title: LoRAShear: Efficient Large Language Model Structured Pruning and
    Knowledge Recovery
summary: Large Language Models (LLMs) have transformed the landscape of artificial
  intelligence, while their enormous size presents significant challenges in
  terms of computational costs. We introduce LoRAShear, a novel efficient
  approach to structurally prune LLMs and recover knowledge. Given general LLMs,
  LoRAShear first creates the dependency graphs to discover minimally removal
  structures and analyze the knowledge distribution. It then proceeds progressive
  structured pruning on LoRA adaptors and enables inherent knowledge transfer to
  better preserve the information in the redundant structures. To recover the
  lost knowledge during pruning, LoRAShear meticulously studies and proposes a
  dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow
  down the performance gap to the full models. Numerical results demonstrate that
  by only using one GPU within a couple of GPU days, LoRAShear effectively
  reduced footprint of LLMs by 20% with only 1.0% performance degradation and
  significantly outperforms state-of-the-arts. The source code will be available
  at https://github.com/microsoft/lorashear.
paper_authors._id: 654089d31c4b203c012e6da6
paper_authors.name: Tianyi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T17:02:01.315Z
paper_authors._id: 64ad94f05a4a60156925ec96
paper_authors.avatarUrl: /avatars/643bdb076e703bfcc89cec6fccb756c6.svg
paper_authors.fullname: Tianyi Chen
paper_authors.type: user
paper_authors.user: tianyic
paper_authors._id: 654089d31c4b203c012e6da7
paper_authors.name: Tianyu Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T08:24:35.618Z
paper_authors._id: 646b0c6d3e2a7b0659452fc7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b0c6d3e2a7b0659452fc7/aqT9nbrJtOk38tDw_12xW.jpeg
paper_authors.fullname: Tianyu Ding
paper_authors.type: user
paper_authors.user: tding1
paper_authors._id: 654089d31c4b203c012e6da8
paper_authors.name: Badal Yadav
paper_authors._id: 654089d31c4b203c012e6da9
paper_authors.name: Ilya Zharkov
paper_authors._id: 654089d31c4b203c012e6daa
paper_authors.name: Luming Liang
paper_title: LoRAShear: Efficient Large Language Model Structured Pruning and
    Knowledge Recovery
paper_summary: Large Language Models (LLMs) have transformed the landscape of artificial
  intelligence, while their enormous size presents significant challenges in
  terms of computational costs. We introduce LoRAShear, a novel efficient
  approach to structurally prune LLMs and recover knowledge. Given general LLMs,
  LoRAShear first creates the dependency graphs to discover minimally removal
  structures and analyze the knowledge distribution. It then proceeds progressive
  structured pruning on LoRA adaptors and enables inherent knowledge transfer to
  better preserve the information in the redundant structures. To recover the
  lost knowledge during pruning, LoRAShear meticulously studies and proposes a
  dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow
  down the performance gap to the full models. Numerical results demonstrate that
  by only using one GPU within a couple of GPU days, LoRAShear effectively
  reduced footprint of LLMs by 20% with only 1.0% performance degradation and
  significantly outperforms state-of-the-arts. The source code will be available
  at https://github.com/microsoft/lorashear.
paper_ai_keywords: Large Language Models (LLMs) LoRAShear dependency graphs node structures knowledge distribution structured pruning LoRA adaptors knowledge transfer dynamic fine-tuning dynamic data adaptors performance gap GPU