ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect
As Large Language Models (LLMs) continue to advance in performance, their
size has escalated significantly, with current LLMs containing billions or even
trillions of parameters. However, in this study, we discovered that many layers
of LLMs exhibit high similarity, and some layers play a negligible role in
network functionality. Based on this observation, we define a metric called
Block Influence (BI) to gauge the significance of each layer in LLMs. We then
propose a straightforward pruning approach: layer removal, in which we directly
delete the redundant layers in LLMs based on their BI scores. Experiments
demonstrate that our method, which we call ShortGPT, significantly outperforms
previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT
is orthogonal to quantization-like methods, enabling further reduction in
parameters and computation. The ability to achieve better results through
simple layer removal, as opposed to more complex pruning techniques, suggests a
high degree of redundancy in the model architecture.
65e94d460788949d203756db
Xin Men
65e94d460788949d203756dc
Mingyu Xu
65e94d460788949d203756dd
Qingyu Zhang
claimed_verified
2025-04-03T08:30:21.269Z
67014d33126f9ab39fc52481
/avatars/60d1f791e7f3201ce1aef72e9216ff78.svg
Qianhao Yuan
user
yuanqianhao
65e94d460788949d203756de
Bingning Wang
claimed_verified
2024-03-08T08:33:41.942Z
640dd3e0c364a086c6322ad2
https://cdn-avatars.huggingface.co/v1/production/uploads/640dd3e0c364a086c6322ad2/z11CvZcbWGSRtWXzvem6I.png
Daniel Wang
user
DanielWang
65e94d460788949d203756df
Hongyu Lin
65e94d460788949d203756e0
Yaojie Lu
claimed_verified
2025-01-08T13:25:17.611Z
6216496a9b34d2fb49144599
https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
Yaojie Lu
user
luyaojie
65e94d460788949d203756e1
Xianpei Han
65e94d460788949d203756e2
Weipeng Chen
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect
As Large Language Models (LLMs) continue to advance in performance, their
size has escalated significantly, with current LLMs containing billions or even
trillions of parameters. However, in this study, we discovered that many layers
of LLMs exhibit high similarity, and some layers play a negligible role in
network functionality. Based on this observation, we define a metric called
Block Influence (BI) to gauge the significance of each layer in LLMs. We then
propose a straightforward pruning approach: layer removal, in which we directly
delete the redundant layers in LLMs based on their BI scores. Experiments
demonstrate that our method, which we call ShortGPT, significantly outperforms
previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT
is orthogonal to quantization-like methods, enabling further reduction in
parameters and computation. The ability to achieve better results through
simple layer removal, as opposed to more complex pruning techniques, suggests a
high degree of redundancy in the model architecture.
A new layer pruning method called ShortGPT significantly improves model pruning performance by removing redundant layers based on Block Influence scores.
Large Language Models Block Influence (BI) layer removal model pruning ShortGPT state-of-the-art (SOTA) quantization-like methods redundancy model architecture