title: ShortGPT: Layers in Large Language Models are More Redundant Than You
    Expect
summary: As Large Language Models (LLMs) continue to advance in performance, their
  size has escalated significantly, with current LLMs containing billions or even
  trillions of parameters. However, in this study, we discovered that many layers
  of LLMs exhibit high similarity, and some layers play a negligible role in
  network functionality. Based on this observation, we define a metric called
  Block Influence (BI) to gauge the significance of each layer in LLMs. We then
  propose a straightforward pruning approach: layer removal, in which we directly
  delete the redundant layers in LLMs based on their BI scores. Experiments
  demonstrate that our method, which we call ShortGPT, significantly outperforms
  previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT
  is orthogonal to quantization-like methods, enabling further reduction in
  parameters and computation. The ability to achieve better results through
  simple layer removal, as opposed to more complex pruning techniques, suggests a
  high degree of redundancy in the model architecture.
paper_authors._id: 65e94d460788949d203756db
paper_authors.name: Xin Men
paper_authors._id: 65e94d460788949d203756dc
paper_authors.name: Mingyu Xu
paper_authors._id: 65e94d460788949d203756dd
paper_authors.name: Qingyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:30:21.269Z
paper_authors._id: 67014d33126f9ab39fc52481
paper_authors.avatarUrl: /avatars/60d1f791e7f3201ce1aef72e9216ff78.svg
paper_authors.fullname: Qianhao Yuan
paper_authors.type: user
paper_authors.user: yuanqianhao
paper_authors._id: 65e94d460788949d203756de
paper_authors.name: Bingning Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T08:33:41.942Z
paper_authors._id: 640dd3e0c364a086c6322ad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dd3e0c364a086c6322ad2/z11CvZcbWGSRtWXzvem6I.png
paper_authors.fullname: Daniel Wang
paper_authors.type: user
paper_authors.user: DanielWang
paper_authors._id: 65e94d460788949d203756df
paper_authors.name: Hongyu Lin
paper_authors._id: 65e94d460788949d203756e0
paper_authors.name: Yaojie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T13:25:17.611Z
paper_authors._id: 6216496a9b34d2fb49144599
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
paper_authors.fullname: Yaojie Lu
paper_authors.type: user
paper_authors.user: luyaojie
paper_authors._id: 65e94d460788949d203756e1
paper_authors.name: Xianpei Han
paper_authors._id: 65e94d460788949d203756e2
paper_authors.name: Weipeng Chen
paper_title: ShortGPT: Layers in Large Language Models are More Redundant Than You
    Expect
paper_summary: As Large Language Models (LLMs) continue to advance in performance, their
  size has escalated significantly, with current LLMs containing billions or even
  trillions of parameters. However, in this study, we discovered that many layers
  of LLMs exhibit high similarity, and some layers play a negligible role in
  network functionality. Based on this observation, we define a metric called
  Block Influence (BI) to gauge the significance of each layer in LLMs. We then
  propose a straightforward pruning approach: layer removal, in which we directly
  delete the redundant layers in LLMs based on their BI scores. Experiments
  demonstrate that our method, which we call ShortGPT, significantly outperforms
  previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT
  is orthogonal to quantization-like methods, enabling further reduction in
  parameters and computation. The ability to achieve better results through
  simple layer removal, as opposed to more complex pruning techniques, suggests a
  high degree of redundancy in the model architecture.
paper_ai_summary: A new layer pruning method called ShortGPT significantly improves model pruning performance by removing redundant layers based on Block Influence scores.
paper_ai_keywords: Large Language Models Block Influence (BI) layer removal model pruning ShortGPT state-of-the-art (SOTA) quantization-like methods redundancy model architecture