title: SEAP: Training-free Sparse Expert Activation Pruning Unlock the
    Brainpower of Large Language Models
summary: Large Language Models have achieved remarkable success across various natural
  language processing tasks, yet their high computational cost during inference
  remains a major bottleneck. This paper introduces Sparse Expert Activation
  Pruning (SEAP), a training-free pruning method that selectively retains
  task-relevant parameters to reduce inference overhead. Inspired by the
  clustering patterns of hidden states and activations in LLMs, SEAP identifies
  task-specific expert activation patterns and prunes the model while preserving
  task performance and enhancing computational efficiency. Experimental results
  demonstrate that SEAP significantly reduces computational overhead while
  maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both
  WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2%
  performance drop compared to the dense model. These findings highlight SEAP's
  scalability and effectiveness, making it a promising approach for optimizing
  large-scale LLMs.
paper_authors._id: 67cfa0c1edb742caa3572982
paper_authors.name: Xun Liang
paper_authors._id: 67cfa0c1edb742caa3572983
paper_authors.name: Hanyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:22:46.104Z
paper_authors._id: 669e0b93c7cb0568dac6e92e
paper_authors.avatarUrl: /avatars/a39ea77d7391f164af8a80f94f85f2ca.svg
paper_authors.fullname: hanyu Wang
paper_authors.type: user
paper_authors.user: UglyToilet
paper_authors._id: 67cfa0c1edb742caa3572984
paper_authors.name: Huayi Lai
paper_authors._id: 67cfa0c1edb742caa3572985
paper_authors.name: Simin Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T10:32:27.196Z
paper_authors._id: 66daea8776dbaaa372eabec5
paper_authors.avatarUrl: /avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
paper_authors.fullname: siminniu
paper_authors.type: user
paper_authors.user: siminniu
paper_authors._id: 67cfa0c1edb742caa3572986
paper_authors.name: Shichao Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T10:32:34.461Z
paper_authors._id: 656f339a5273668d5b946b33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656f339a5273668d5b946b33/o2nBvQiOKKP5IfDmnpHP2.jpeg
paper_authors.fullname: Shichao Song
paper_authors.type: user
paper_authors.user: Ki-Seki
paper_authors._id: 67cfa0c1edb742caa3572987
paper_authors.name: Jiawei Yang
paper_authors._id: 67cfa0c1edb742caa3572988
paper_authors.name: Jihao Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:24:39.537Z
paper_authors._id: 658e85bb5b7553ca5c29ba89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
paper_authors.fullname: Jihao Zhao
paper_authors.type: user
paper_authors.user: Robot2050
paper_authors._id: 67cfa0c1edb742caa3572989
paper_authors.name: Feiyu Xiong
paper_authors._id: 67cfa0c1edb742caa357298a
paper_authors.name: Bo Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T10:43:43.972Z
paper_authors._id: 66b02a2642c34e7a212133c0
paper_authors.avatarUrl: /avatars/737a69b095e8c427ecd08f870b173635.svg
paper_authors.fullname: Bo Tang
paper_authors.type: user
paper_authors.user: BO1022
paper_authors._id: 67cfa0c1edb742caa357298b
paper_authors.name: Zhiyu Li
paper_title: SEAP: Training-free Sparse Expert Activation Pruning Unlock the
    Brainpower of Large Language Models
paper_summary: Large Language Models have achieved remarkable success across various natural
  language processing tasks, yet their high computational cost during inference
  remains a major bottleneck. This paper introduces Sparse Expert Activation
  Pruning (SEAP), a training-free pruning method that selectively retains
  task-relevant parameters to reduce inference overhead. Inspired by the
  clustering patterns of hidden states and activations in LLMs, SEAP identifies
  task-specific expert activation patterns and prunes the model while preserving
  task performance and enhancing computational efficiency. Experimental results
  demonstrate that SEAP significantly reduces computational overhead while
  maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both
  WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2%
  performance drop compared to the dense model. These findings highlight SEAP's
  scalability and effectiveness, making it a promising approach for optimizing
  large-scale LLMs.
paper_ai_keywords: Sparse Expert Activation Pruning (SEAP) hidden states activations task-specific expert activation patterns computational efficiency