title: Optimal Brain Apoptosis
summary: The increasing complexity and parameter count of Convolutional Neural
  Networks (CNNs) and Transformers pose challenges in terms of computational
  efficiency and resource demands. Pruning has been identified as an effective
  strategy to address these challenges by removing redundant elements such as
  neurons, channels, or connections, thereby enhancing computational efficiency
  without heavily compromising performance. This paper builds on the foundational
  work of Optimal Brain Damage (OBD) by advancing the methodology of parameter
  importance estimation using the Hessian matrix. Unlike previous approaches that
  rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel
  pruning method that calculates the Hessian-vector product value directly for
  each parameter. By decomposing the Hessian matrix across network layers and
  identifying conditions under which inter-layer Hessian submatrices are
  non-zero, we propose a highly efficient technique for computing the
  second-order Taylor expansion of parameters. This approach allows for a more
  precise pruning process, particularly in the context of CNNs and Transformers,
  as validated in our experiments including VGG19, ResNet32, ResNet50, and
  ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at
  https://github.com/NEU-REAL/OBA.
paper_authors._id: 67c59a7e6eb050aa82406452
paper_authors.name: Mingyuan Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T16:07:21.192Z
paper_authors._id: 668e62f6514c46e257387f6b
paper_authors.avatarUrl: /avatars/601b111141141cb2ea710b3166e62cd0.svg
paper_authors.fullname: Mingyuan Sun
paper_authors.type: user
paper_authors.user: mingyuansun
paper_authors._id: 67c59a7e6eb050aa82406453
paper_authors.name: Zheng Fang
paper_authors._id: 67c59a7e6eb050aa82406454
paper_authors.name: Jiaxu Wang
paper_authors._id: 67c59a7e6eb050aa82406455
paper_authors.name: Junjie Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:36:31.337Z
paper_authors._id: 6818d92ba73377008d118813
paper_authors.avatarUrl: /avatars/03d3efa5472e6e4bb025cf4345a2dfd4.svg
paper_authors.fullname: junjie jiang
paper_authors.type: user
paper_authors.user: joyjoyjoy3
paper_authors._id: 67c59a7e6eb050aa82406456
paper_authors.name: Delei Kong
paper_authors._id: 67c59a7e6eb050aa82406457
paper_authors.name: Chenming Hu
paper_authors._id: 67c59a7e6eb050aa82406458
paper_authors.name: Yuetong Fang
paper_authors._id: 67c59a7e6eb050aa82406459
paper_authors.name: Renjing Xu
paper_title: Optimal Brain Apoptosis
paper_summary: The increasing complexity and parameter count of Convolutional Neural
  Networks (CNNs) and Transformers pose challenges in terms of computational
  efficiency and resource demands. Pruning has been identified as an effective
  strategy to address these challenges by removing redundant elements such as
  neurons, channels, or connections, thereby enhancing computational efficiency
  without heavily compromising performance. This paper builds on the foundational
  work of Optimal Brain Damage (OBD) by advancing the methodology of parameter
  importance estimation using the Hessian matrix. Unlike previous approaches that
  rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel
  pruning method that calculates the Hessian-vector product value directly for
  each parameter. By decomposing the Hessian matrix across network layers and
  identifying conditions under which inter-layer Hessian submatrices are
  non-zero, we propose a highly efficient technique for computing the
  second-order Taylor expansion of parameters. This approach allows for a more
  precise pruning process, particularly in the context of CNNs and Transformers,
  as validated in our experiments including VGG19, ResNet32, ResNet50, and
  ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at
  https://github.com/NEU-REAL/OBA.
paper_ai_summary: The paper introduces Optimal Brain Apoptosis (OBA), a novel pruning method that precisely estimates parameter importance using Hessian-vector products, enhancing computational efficiency in CNNs and Transformers.
paper_ai_keywords: Convolutional Neural Networks Transformers pruning Optimal Brain Damage Hessian matrix Hessian-vector product parameter importance second-order Taylor expansion VGG19 ResNet32 ResNet50 ViT-B/16 CIFAR10 CIFAR100 Imagenet