SEAP: Training-free Sparse Expert Activation Pruning Unlock the
  Brainpower of Large Language Models
Large Language Models have achieved remarkable success across various natural
language processing tasks, yet their high computational cost during inference
remains a major bottleneck. This paper introduces Sparse Expert Activation
Pruning (SEAP), a training-free pruning method that selectively retains
task-relevant parameters to reduce inference overhead. Inspired by the
clustering patterns of hidden states and activations in LLMs, SEAP identifies
task-specific expert activation patterns and prunes the model while preserving
task performance and enhancing computational efficiency. Experimental results
demonstrate that SEAP significantly reduces computational overhead while
maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both
WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2%
performance drop compared to the dense model. These findings highlight SEAP's
scalability and effectiveness, making it a promising approach for optimizing
large-scale LLMs.
67cfa0c1edb742caa3572982
Xun Liang
67cfa0c1edb742caa3572983
Hanyu Wang
claimed_verified
2025-03-11T08:22:46.104Z
669e0b93c7cb0568dac6e92e
/avatars/a39ea77d7391f164af8a80f94f85f2ca.svg
hanyu Wang
user
UglyToilet
67cfa0c1edb742caa3572984
Huayi Lai
67cfa0c1edb742caa3572985
Simin Niu
admin_assigned
2025-03-11T10:32:27.196Z
66daea8776dbaaa372eabec5
/avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
siminniu
user
siminniu
67cfa0c1edb742caa3572986
Shichao Song
admin_assigned
2025-03-11T10:32:34.461Z
656f339a5273668d5b946b33
https://cdn-avatars.huggingface.co/v1/production/uploads/656f339a5273668d5b946b33/o2nBvQiOKKP5IfDmnpHP2.jpeg
Shichao Song
user
Ki-Seki
67cfa0c1edb742caa3572987
Jiawei Yang
67cfa0c1edb742caa3572988
Jihao Zhao
claimed_verified
2025-03-13T08:24:39.537Z
658e85bb5b7553ca5c29ba89
https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
Jihao Zhao
user
Robot2050
67cfa0c1edb742caa3572989
Feiyu Xiong
67cfa0c1edb742caa357298a
Bo Tang
admin_assigned
2025-03-11T10:43:43.972Z
66b02a2642c34e7a212133c0
/avatars/737a69b095e8c427ecd08f870b173635.svg
Bo Tang
user
BO1022
67cfa0c1edb742caa357298b
Zhiyu Li
SEAP: Training-free Sparse Expert Activation Pruning Unlock the
  Brainpower of Large Language Models
Large Language Models have achieved remarkable success across various natural
language processing tasks, yet their high computational cost during inference
remains a major bottleneck. This paper introduces Sparse Expert Activation
Pruning (SEAP), a training-free pruning method that selectively retains
task-relevant parameters to reduce inference overhead. Inspired by the
clustering patterns of hidden states and activations in LLMs, SEAP identifies
task-specific expert activation patterns and prunes the model while preserving
task performance and enhancing computational efficiency. Experimental results
demonstrate that SEAP significantly reduces computational overhead while
maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both
WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2%
performance drop compared to the dense model. These findings highlight SEAP's
scalability and effectiveness, making it a promising approach for optimizing
large-scale LLMs.
Sparse Expert Activation Pruning (SEAP) is a method for pruning large language models that reduces computational overhead while maintaining accuracy by identifying and retaining task-specific expert activations.
Sparse Expert Activation Pruning SEAP large language models computational overhead task-specific expert activation patterns