Optimal Brain Apoptosis
The increasing complexity and parameter count of Convolutional Neural
Networks (CNNs) and Transformers pose challenges in terms of computational
efficiency and resource demands. Pruning has been identified as an effective
strategy to address these challenges by removing redundant elements such as
neurons, channels, or connections, thereby enhancing computational efficiency
without heavily compromising performance. This paper builds on the foundational
work of Optimal Brain Damage (OBD) by advancing the methodology of parameter
importance estimation using the Hessian matrix. Unlike previous approaches that
rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel
pruning method that calculates the Hessian-vector product value directly for
each parameter. By decomposing the Hessian matrix across network layers and
identifying conditions under which inter-layer Hessian submatrices are
non-zero, we propose a highly efficient technique for computing the
second-order Taylor expansion of parameters. This approach allows for a more
precise pruning process, particularly in the context of CNNs and Transformers,
as validated in our experiments including VGG19, ResNet32, ResNet50, and
ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at
https://github.com/NEU-REAL/OBA.
67c59a7e6eb050aa82406452
Mingyuan Sun
claimed_verified
2025-03-03T16:07:21.192Z
668e62f6514c46e257387f6b
/avatars/601b111141141cb2ea710b3166e62cd0.svg
Mingyuan Sun
user
mingyuansun
67c59a7e6eb050aa82406453
Zheng Fang
67c59a7e6eb050aa82406454
Jiaxu Wang
67c59a7e6eb050aa82406455
Junjie Jiang
claimed_verified
2025-05-06T08:36:31.337Z
6818d92ba73377008d118813
/avatars/03d3efa5472e6e4bb025cf4345a2dfd4.svg
junjie jiang
user
joyjoyjoy3
67c59a7e6eb050aa82406456
Delei Kong
67c59a7e6eb050aa82406457
Chenming Hu
67c59a7e6eb050aa82406458
Yuetong Fang
67c59a7e6eb050aa82406459
Renjing Xu
Optimal Brain Apoptosis
The increasing complexity and parameter count of Convolutional Neural
Networks (CNNs) and Transformers pose challenges in terms of computational
efficiency and resource demands. Pruning has been identified as an effective
strategy to address these challenges by removing redundant elements such as
neurons, channels, or connections, thereby enhancing computational efficiency
without heavily compromising performance. This paper builds on the foundational
work of Optimal Brain Damage (OBD) by advancing the methodology of parameter
importance estimation using the Hessian matrix. Unlike previous approaches that
rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel
pruning method that calculates the Hessian-vector product value directly for
each parameter. By decomposing the Hessian matrix across network layers and
identifying conditions under which inter-layer Hessian submatrices are
non-zero, we propose a highly efficient technique for computing the
second-order Taylor expansion of parameters. This approach allows for a more
precise pruning process, particularly in the context of CNNs and Transformers,
as validated in our experiments including VGG19, ResNet32, ResNet50, and
ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at
https://github.com/NEU-REAL/OBA.
The paper introduces Optimal Brain Apoptosis (OBA), a novel pruning method that precisely estimates parameter importance using Hessian-vector products, enhancing computational efficiency in CNNs and Transformers.
Convolutional Neural Networks Transformers pruning Optimal Brain Damage Hessian matrix Hessian-vector product parameter importance second-order Taylor expansion VGG19 ResNet32 ResNet50 ViT-B/16 CIFAR10 CIFAR100 Imagenet