## CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution

Zichen Song, Jiakang Li, Songning Lai, Sitan Huang (Co-First Author)

Abstract. Spiking neural networks (SNNs) have shown promise in various dynamic  visual  tasks,  yet  those  ready  for  practical  deployment  often  lack  the compactness and robustness essential in resource-limited and safety-critical settings.  Prior  research  has  predominantly  concentrated  on  enhancing  the  compactness or robustness of artificial neural networks through strategies like network pruning and adversarial training, with little exploration into similar methodologies  for  SNNs.  Robust  pruning  of  SNNs  aims  to  reduce  computational overhead while preserving both accuracy and robustness. Current robust pruning approaches generally necessitate expert knowledge and iterative experimentation to establish suitable pruning criteria or auxiliary modules, thus constraining their broader application. Concurrently, evolutionary algorithms (EAs) have been employed to automate the pruning of artificial neural networks, delivering remarkable outcomes yet overlooking the aspect of robustness. In this work, we propose CCSRP, an innovative robust pruning method for SNNs, underpinned by cooperative co-evolution. Robust pruning is articulated as a tri-objective optimization challenge, striving to balance accuracy, robustness, and compactness concurrently,  resolved  through  a  cooperative  co-evolutionary  pruning  framework  that  independently  prunes  filters  across  layers  using  EAs.  Our  experiments on CIFAR-10 and SVHN demonstrate that CCSRP can match or exceed the performance of the latest methodologies.

Keywords: Model compression ·  Spiking Neural network · Neural network pruning ·  Robustness · Evolutionary algorithm · Cooperative coevolution.

## 1 Introduction

In recent years, spiking neural networks (SNNs) have achieved significant success in the field of dynamic vision, such as spiking neural state image classification and object  detection.  Despite  their  impressive  performance,  the  high  computational  cost associated  with  converting  artificial  neural  networks  (ANNs)  to  SNNs  limits  their deployment in resource-constrained scenarios.[1] Additionally, SNNs are susceptible to malicious attacks, posing a challenge to their reliability in safety-critical environments. Thus, enhancing both the compactness and robustness of SNNs is crucial in many practical applications, such as event cameras.[2]

However, most previous work has focused solely on enhancing either the compactness or robustness of spiking neural networks.[3] On one hand, various model compression techniques have been proposed to reduce the computational cost of SNNs, such as neural network pruning and quantization. Among these, neural network pruning  aims  to  remove  redundant  parameters  in  networks  while  maintaining  accuracy, and has achieved considerable success. On the other hand, methods like adversarial training,  which  aim  to  minimize  training  loss  on  adversarial  examples,  can  significantly enhance the robustness of SNNs.[4]

Recent efforts have considered network robustness in the context of pruning SNNs. Typically, these approaches use expert-designed criteria to measure the importance of network weights and prune accordingly. However, designing and tuning such criteria require  extensive  expertise  and  laborious  experimentation,  making  them difficult  to apply in practical scenarios with diverse datasets and SNN architectures. Moreover, these efforts mainly focus on unstructured neural network pruning, which hardly reduces  computational  costs  in  real-world  applications  due  to  the  resulting  irregular structures  being  incompatible  with  mainstream  software  and  hardware  frameworks. [5]  Thus,  an  automated  structured  robust  pruning  method  is  essential  for  practical applications.  Robust  pruning  of  SNNs  can  naturally  be  framed  as  an  optimization problem, aiming to find a subnet of the original network that maintains high accuracy and robustness but with lower computational cost.[6] Evolutionary algorithms (EAs), inspired by natural evolution, have been used for automatically pruning SNNs. However, unlike artificial neural networks from the last century, modern SNNs typically comprise dozens of layers and millions of parameters, implying a vast search space. For  EAs,  finding  satisfactory  solutions  within  a  limited  computational  overhead  is challenging. Recently, Shang et al. proposed an evolutionary pruning method inspired by cooperative co-evolution, CCEP, which has shown encouraging results for largescale pruning problems. However, their focus was solely on accuracy without considering  robustness.  In  this  paper,  we  introduce  a  novel  Cooperative  Coevolutionary Strategy  for  Robust  Pruning  (CCSRP).[7]  The  robust  pruning  problem  is  explicitly formulated as a three-objective optimization problem, aiming to simultaneously optimize accuracy, robustness, and compactness. A cooperative co-evolution framework is employed to tackle the robust pruning problem, dividing the search space by layer and applying an EA to optimize each group independently. Additionally, to address the time-consuming process of generating adversarial examples for each pruned network, we devise an adversarial example generation method to improve the efficiency of robustness evaluation.[8]

Our contributions are summarized as follows:

- 1. We present a novel framework, CCSRP, that considers network robustness during the pruning process and automatically solves the three-objective robust pruning problem through cooperative co-evolution. To our knowledge, this is the first application of EAs to robust pruning of spiking neural networks.

- 2. We introduce an adversarial example generation method to enhance the efficiency of evaluating the robustness of pruned networks.
- 3. We compare CCSRP with previous methods through experiments on three network architectures  and  two  datasets.  Experimental  results  demonstrate  that  CCSRP  can achieve performance comparable to state-of-the-art methods.

## 2 Related work

## 2.1 Spiking Neural Network Pruning

The objective of pruning in spiking neural networks (SNNs) is to enhance operational efficiency  by  eliminating  unnecessary  components.  Current  pruning  strategies  are predominantly  categorized  into  two  types:  unstructured  and  structured  pruning.[9] Unstructured pruning involves direct adjustments to the weights within the network, theoretically  offering  significant  computational  speed-ups.  However,  the  resulting sparse  matrices  and  discontinuous  structures  are  often  incompatible  with  existing software  and  hardware  environments,  making  practical  acceleration  challenging  to achieve. In contrast, structured pruning targets the systematic removal of components, such as filters in convolutional layers of SNNs, demonstrating superior performance in real-world applications, thereby gaining increased popularity and attention.[10]

Regarding  the  identification  of  redundant  components,  previous  structured  pruning approaches  can  be  broadly  divided  into  criteria-based  and  learning-based  methods.[11] Criteria-based methods rely on expert-designed rules to identify and prune non-essential components, whereas learning-based methods utilize auxiliary modules to  assess  the  importance  of  components  for  subsequent pruning.  Nevertheless,  both methods heavily depend on domain expertise, restricting their widespread application and flexibility.[12]

To reduce the dependency on expert knowledge, employing evolutionary algorithms (EAs) for the automatic discovery of optimized pruned network architectures emerges as a natural solution. Despite this, the vast search space of SNNs presents a significant challenge to EAs.[13] Recently, an innovative pruning method inspired by cooperative co-evolution, named CCEP, has been introduced. It adopts a divide-and-conquer strategy to tackle the immense search space, demonstrating impressive performance and  underscoring  the  substantial  potential  of  EA-based  methods  in  the  domain  of neural network pruning. However, previous EA-based pruning approaches have not taken into account the robustness of the network, a factor of critical importance for many application scenarios.[14]

## 2.2 Robust Spiking Neural Network Pruning

Recent studies have explored the relationship between robustness and network capacity, uncovering that a sub-network of the original network may exhibit similar or even

superior robustness compared to the original network, with considerable variance in robustness  among  different  sub-networks.  This  discovery  has  spurred  research  into robust  pruning  for  spiking  neural  networks  (SNNs),  aiming  to  identify  a  compact SNN that retains robustness.[15] The existing handful of methods typically employ adversarial training to train a network and proceed with unstructured pruning based on criteria designed by experts. For instance, ADV-LWM prunes weights with small l1-norms and fine-tunes the resulting network through adversarial training to regain robustness. Utilizing the ADMM pruning framework, one approach replaces the original training loss with an adversarial counterpart. HYDRA assigns importance scores to all weights within the network, optimizing the adversarial loss by adjusting these scores  while  freezing  the  weights.  Subsequently,  weights  with  minimal  importance scores are pruned. DNR opts to prune filters corresponding to feature matrices with small  Frobenius  norms.  Moreover,  these  methods  necessitate  appropriate  pruning ratios  for  each  layer,  which  often  requires  extensive  expert  knowledge  and  iterative experimentation.[16]

## 3 CCSRP Method

Consider 𝑵 to  be  a  thoroughly  trained  neural  network  composed  of 𝒏 convolution layers,  designated  as 𝑳𝟏, 𝑳 𝟐 , … , 𝑳 𝒏 ,  where  each  layer 𝑳𝒊 contains 𝒍 𝒊 filters  and 𝑳𝒊𝒋 refers to the jth filter of the ith layer. The process of robust pruning in neural networks is an optimization challenge aimed at isolating a group of filters within 𝑵 . The goal is to enhance both the network's accuracy and its robustness against perturbations while concurrently  reducing  the  computational  burden.  We  define  a  mask  vector 𝑀 = {𝑚𝑖𝑗 |𝑚 𝑖𝑗 ∈ {0,1}, 𝑖 ∈ {1,2, … , 𝑛}, 𝑗 ∈ {1,2, … , 𝑙 }} 𝑖 with 𝑚𝑖𝑗 = 1 signifying  the  retention of filter 𝑳𝒊𝒋 .  Therefore, we can represent a pruned network with the mask 𝑴 as follows:

The  performance  of  the  pruned  network 𝑵𝑴 is  measured  by 𝐴𝐶𝐶(𝑵𝑴) ,  indicating accuracy on standard datasets, and 𝐴𝐶𝐶𝑟(𝑵𝑴) , indicating robustness as measured by accuracy  against  adversarial  examples.  Additionally, 𝐹𝐿𝑂𝑃 𝑠 (𝑵𝑴) accounts  for  the floating-point operations count, which assesses the computational expenses. We pose the robust pruning task as:

Given that a Spiking Neural Networks (SNNs) can have a substantial quantity of filters eligible for pruning, identified by ∑ 𝑛 𝑙 𝑖 𝑖=1 , this scenario poses a substantial optimization conundrum. To address this, we introduce CCSRP, a cutting-edge method for robust pruning. Drawing on the concepts from CCEP and CCRP, we utilize a cooperative  coevolution-based  pruning  framework.  This  stratagem  segments  the  search

space into individual layers and applies evolutionary algorithms (EAs) to each layer in isolation. We prioritize robust accuracy as the main optimization criterion, steering the pruning process of neural networks in the direction of robustness. It's noteworthy to mention that evaluating 𝐴𝐶𝐶𝑟 (𝑵𝑴 ) can be labor-intensive, as it requires generating bespoke  adversarial  examples  for  each  pruned  network  variant.  To  circumvent  this issue, we suggest an adversarial example generation technique that obviates the need for producing such examples for each pruned network iteration.[17]

## 3.1 EA

Algorithm  1  delineates  the  procedure  of  the  CCSRP  framework.  This  framework refines a well-trained neural network through iterative pruning, ultimately yielding a suite of pruned networks that maintain robustness for selection. The iterative process is  articulated  as  follows.  Initially,  the  algorithm  generates  a  mask 𝑴 tailored  to  the network designated for pruning. Subsequently, this mask 𝑴 is partitioned into n segments corresponding to the hierarchical layers of the network. Following this, a set of adversarial  examples 𝑫𝒂 is  produced  for  assessing  the  efficacy  of  the  pruned  network.  For  each  segmented  group,  an  evolutionary  algorithm  (EA)  is  employed  to orchestrate optimization, procuring 𝑚𝑖 ′ ,  indicative of the pruning outcome for the ith layer. By amalgamating 𝑚𝑖 ′ for all n layers and applying them to the baseline network Nb, the pruned network N' is procured. Post-pruning, to recoup the network's accuracy and robust accuracy, the pruned network N' undergoes fine-tuning via adversarial training. The fine-tuned model is then established as the new baseline network Nb for subsequent pruning in the next iteration and is archived in H . [18]

After  T  iterations,  the  CCSRP  framework  ceases  and  returns  the  pruned  networks stored in archive H. An illustrative depiction of the CCSRP framework is also exhibited in Figure 1. Within each group of the evolutionary algorithm (EA) process, we engage  a  quintessential  evolutionary  procedure  that  commences  with  the  random generation of an initial subpopulation. New individuals are bred by applying reproductive operators, followed by the evaluation of fitness for each, and the selection of the  fittest  individuals  to  advance  to  the  succeeding  generation.  Upon  reaching  the termination criterion, the EA selects an individual from the ultimate subpopulation, which epitomizes the pruned layer for that group.[19]

The EA process within each group is elucidated in detail. Initially, it commences by generating an initial subpopulation P consisting of d individuals. An individual |𝑚|0 , with all bits set to 1, is incorporated into P to foster conservative pruning strategies. The remaining d-1 individuals are generated using a modified bitwise mutation operator with a mutation rate 𝑝1 .  During each EA generation, d new progenies are engendered by randomly choosing d individuals from the subpopulation with replacement, followed by the application of a bitwise mutation operator with a mutation rate 𝑝2 . In accordance with CCEP, we have adapted the standard bitwise mutation operator to mitigate overly aggressive pruning. Specifically, a ratio bound r constrains the number of filters to be pruned. [20]

## Algorithm 1. Combined CCRP with EA Optimization

Input: A well trained SNNs N with n layers, maximum number T of iterations, training; set Dt, a randomly sampled part Ds of the training set, adversarial data set Da, population size d, mutation rate p1, p2, ratio bound r, maximum number G of generations

Output: A set of pruned networks with different sizes

∅

| 1: 2:   | Let H = , i = 0; Set base network Nb = N;                                                                                           |
|---------|-------------------------------------------------------------------------------------------------------------------------------------|
| 3:      | while i < T do                                                                                                                      |
| 4:      | Generate a mask M based on Nb and initialize it with all bits equal to 1;                                                           |
| 5:      | for each layer l from 1 to n do Set mi as the lth segment of M;                                                                     |
| 6: 7:   | Let j = 0, m0 = mi;                                                                                                                 |
| 8:      | Initialize a subpopulation P with m0 and d-1 individuals generated from m0 by applying the bitwise mutation operator with p1 and r; |
| 9:      | while j < G do                                                                                                                      |
| 10:     | Uniformly randomly select d individuals from P with replacement as the parent individuals;                                          |
| 11:     | Generate d offspring individuals by apply ing the bit-wise mutation operator with p2 and r on each parent individual;               |
| 12:     | Calculate the ACC and ACCr of d offspring individuals by using Ds and Da; the FLOPs of d offspring indi viduals;                    |
| 14:     | Set Q as the union of P and d offspring individuals;                                                                                |
| 15:     | Rank the 2d individuals in Q in descending order by (ACC+ACCr)/2;                                                                   |
| 16:     | Replace the individuals in P with the top d individuals in Q;                                                                       |
| 17:     | j = j + 1;                                                                                                                          |
| 18:     | end while                                                                                                                           |
| 19:     | Select the rank one individual in P as m'l;                                                                                         |
| 20:     | end for                                                                                                                             |
| 21:     | Construct new network N' by combining the m'n;                                                                                      |
| 22:     | Fine-tune N' with Dt; Nb = N';                                                                                                      |
| 24:     | H = H ∪ Nb; i = i + 1;                                                                                                              |
| 26:     | end while                                                                                                                           |
|         | return H                                                                                                                            |
| 27:     |                                                                                                                                     |

Fitness evaluation for an offspring individual, given that each corresponds to a single pruned layer, involves splicing this layer with the other layers from the base network Nb to  form a complete network. This allows for the assessment of accuracy, robust accuracy,  and  FLOPs  of  the  offspring  individuals.  Specifically,  accuracy  (ACC)  is gauged  on  the  clean  dataset 𝐷𝑠 ,  sampled  randomly  from  the  training  set 𝐷𝑡 ;  robust accuracy (ACCr) is evaluated on the adversarial dataset 𝐷𝑎 ; and FLOPs are calculated directly.[21] Following evaluation, the d offspring and the d individuals in the current subpopulation P are  amalgamated  into  a  collective Q .  Considering  the  three  objectives, ranking individuals presents a complexity. For ease, individuals in Q are ranked in descending order based on the average of ACC and ACCr. Should there be a tie in average values, the individual with fewer FLOPs is favored. Alternative techniques, such as non-dominated sorting, may be employed and will be explored in future work. After G generations of evolution, the top-ranked individual from the final subpopulation is selected as the pruned outcome for the respective group.[22]

Fig 1. Illustration of the framework of CCSRP.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00794v1-figure-5ba5e7ac-7a6b-4b34-a015-409f2154a4e5.jpg)

## 3.2 Robustness and Comparison

Typically, the robustness of spiking neural networks is gauged by their resilience to adversarial attacks. In this study, we define robust accuracy (ACCr) as the measure of robustness, determined by the network's performance against crafted adversarial examples.[23]  We  employ  the  advanced  PGD  white-box  attack  algorithm  to  generate these examples, noted for its iterative and time-intensive nature.[24] To circumvent the  prohibitive  computational  expense  of  generating  unique  adversarial  samples  for each pruned network, we have devised an efficient generation method. This technique produces a shared adversarial dataset Da during a single iteration of CCSRP by apply-

ing a mutation operation to select layers of the base network Nb and utilizing PGD. This procedure is independently replicated to compile Da , which assesses the robustness of all pruned networks during the current CCRP cycle.[25] Employing various subsets of Nb allows for a more accurate evaluation of a pruned network's robustness. In  comparing  CCSRP  with  CCRP,  the  latter  expands  upon  spiking  neural  network pruning by factoring in robustness, emphasizing not only accuracy and compactness but  also  robustness  as  an  optimization  objective.  CCSRP  integrates  an  adversarial example generation method to minimize the costs of robustness evaluation and incorporates adversarial training in the fine-tuning stage to preserve the pruned network's robustness.[26]

## Algorithm 2: Adversarial Generating

## Input:

- - Base network Nb with n layers, representing the original neural network before pruning.
- -  A  randomly  sampled  part  Ds  of  the  training  set  Dt, which is a subset of the original dataset used for generating adversarial examples.
- - The number k of sampled sub-nets, indicating how many times the generation process is to be repeated to produce diverse adversarial examples.

## Output:

- - Adversarial dataset Da, a collection of adversarial examples used for testing the robustness of the neural network.
- 1: Initialize the adversarial dataset Da as an empty set and the counter i as 0.
- 2: Begin a loop that will iterate k times, where k is the number of adversarial examples sets you wish to generate.
- 3:  Within  each  iteration,  randomly select ⌈ n/k   layers ⌉ from the base network Nb.
- 4: Apply a mutation operation with parameters p1 and r to these selected layers to obtain a sub-net N'.
- 5: Use the  Projected  Gradient  Descent  (PGD)  attack  on this sub-net N' using the sampled dataset Ds to generate a set of adversarial examples A.
- 6: Add the newly generated adversarial examples A to the adversarial dataset Da.
- 7: Increment the counter i by 1.
- 8: Repeat the steps from 3 to 7 until i equals k.
- 9: Once all iterations are complete, return the Da.

End of Algorithm.

## 4 Experiments and Results

Our experimental investigation is conducted in three dimensions. Initially, we compare  the  performance  of  CCSRP  (Cooperative  Coevolutionary  Spiking  Neural  Network Pruning) with the cutting-edge CCRP approach. Subsequently, we expand several prevalent unstructured and structured pruning methods to include robust pruning and juxtapose them with CCSRP. The third dimension involves conducting iterative experiments to evaluate the consistency of CCSRP and the visualization of the pruned network  architectures.  This  evaluation  employs  two  renowned  image  classification datasets: CIFAR-10 and SVHN, as well as three prototype neural networks adapted into their spiking versions: SVGG, SResNet, and SWRN. [27]

Table 1. Comparison ACC, ACCr, and inference speed with unstructured robust pruning.

| Dataset   | Model    | Method   |   Base ACC (%) |   Base ACCr (%) |   Speed (imag- es/s) |
|-----------|----------|----------|----------------|-----------------|----------------------|
| CIFAR-10  |          | ADV-LWM  |          82.52 |           51.91 |              2082.13 |
|           |          | ADV-ADMM |          78.36 |           47.07 |              2114.77 |
|           | VGG-16   | HYDRA    |          82.73 |           51.93 |              2077.57 |
|           |          | CCRP     |          81.32 |           61.34 |              6842.39 |
|           |          | CCSRP    |          82.94 |           62.41 |              5873.12 |
|           |          | ADV-LWM  |          85.6  |           57.2  |              4142.74 |
|           |          | ADV-ADMM |          78.22 |           51.56 |              4375.58 |
|           | WRN-28-4 | HYDRA    |          85.6  |           57.2  |              4016.55 |
|           |          | CCRP     |          85.91 |           53.42 |              4737.09 |
|           |          | CCSRP    |          85.91 |           53.41 |              5823.12 |
| SVHN      |          | ADV-LWM  |          90.5  |           53.5  |              2308.65 |
| SVHN      |          | ADV-ADMM |          89.35 |           54.61 |              2322.72 |
| SVHN      | VGG-16   | HYDRA    |          90.5  |           53.5  |              2334.29 |
|           |          | CCRP     |          86.86 |           53.18 |             11124.6  |
|           |          | CCSRP    |          87.14 |           54.44 |              5973.12 |
|           |          | ADV-LWM  |          93.5  |           60.1  |              5259.51 |
|           |          | ADV-ADMM |          92.14 |           59.07 |              5482.91 |
|           | WRN-28-4 | HYDRA    |          93.5  |           60.1  |              5294.31 |
|           |          | CCRP     |          90.07 |           57.47 |              6467.55 |
|           |          | CCSRP    |          89.98 |           56.41 |              5898.11 |

Adhering to standard filter pruning protocols, CCSRP targets all convolutional layers in  SVGG  and  the  first  convolutional  layer  in  the  residual  blocks  of  SResNet  and SWRN  for  pruning.  The  popular  adversarial  training  method  TRADES  is  applied during both the pre-training and fine-tuning phases. CCSRP's parameter settings are as follows: it operates for 16 iterations (T=16), with the evolutionary algorithm (EA) for each group having a population size d of 5, mutation rates p1 and p2 set at 0.05 and 0.1 respectively, a ratio bound r of 0.1, a maximum of 10 generations G, and Ds constructed by randomly sampling 10% of the training set. When generating adversarial instances, the algorithm samples 5 sub-networks, with k fixed at 5.[28]

Table 2. Comparison ACC, ACCr, and pruning ratio with structured robust pruning.

| Dataset   | Model     | Method         | Base ACC (%)   | Base ACCr (%)           | ACCl (%)        | ACCr (%)          | FLOPs (%)         |
|-----------|-----------|----------------|----------------|-------------------------|-----------------|-------------------|-------------------|
| CIFAR -10 | VGG-16    | L1 HRank       | 81.57 81.91    | 61.71 61.11             | 2.00 7.05       | 3.37 3.01         | 69.23             |
|           |           | CCSRP L1       | 82.57 80.31    | 61.61                   | 0.15 2.47       | 6.32              | 65.85 77.84       |
|           | ResNet-56 | HRank CCSRP L1 | 80.31 81.32    | 48.95 48.95 48.85 53.61 | 0.13 0.24       | -4.62 -2.22 -8.34 | 68.53 50.02 72.32 |
|           | WRN-28-4  | CCSRP          | 85.91 86.11    | 53.18 54.53             | 2.00 -0.89 2.17 | 3.37 -8.26        | 69.23 66.98       |
|           | VGG-16    | L1 HRank       | 86.86          | 53.71                   | 0.40            |                   |                   |
|           |           |                |                |                         |                 | 4.35              | 85.88             |
|           |           |                |                |                         |                 | 5.03              | 65.85             |
|           |           |                | 86.06          |                         |                 |                   |                   |
|           |           | CCSRP          | 87.16          | 53.43                   | -0.96           | 2.64              | 80.54             |
| SVHN      |           | L1             | 85.91          | 52.24                   | -2.04           | -1.78             | 60.08             |
| SVHN      | ResNet-56 | HRank          | 87.09          | 55.57                   | -1.53           | 3.25              | 50.02             |
| SVHN      |           | CCSRP          | 86.91          | 52.32                   | -1.93           | -2.11             | 70.78             |
| SVHN      |           | L1             | 90.07          | 57.47                   | 1.45            | 4.19              |                   |
| SVHN      | WRN-28-4  |                |                |                         |                 |                   | 72.11             |
| SVHN      |           | CCSRP          | 90.47          | 57.34                   | -1.24           | 1.56              | 71.23             |

Under the adversarial training framework employing TRADES, a standard configuration  was adopted. Stochastic Gradient Descent (SGD) was chosen as the optimizer, with an initial learning rate set at 0.1, which was adjusted through a cosine annealing strategy during the fine-tuning phase. The weight decay coefficient was set at 0.0001, with a momentum value of 0.9. Each cycle of the fine-tuning process consisted of 30 training epochs, with a batch size of 128. During the PGD adversarial attack phase, the  adversarial  training  stage  was  configured  with  an  perturbation  budget,  iteration steps, and per-step perturbation values of 8/255, 10, and 2/255, respectively; for evaluation and testing phases, these values were adjusted to 8/255, 40, and 2/255. A comparative analysis of the CCSRP Performance Relative to three cutting-edge unstructured robust pruning techniques-ADV-LWM, ADV-ADMM, and HYDRA-as well as  two  structured  pruning  techniques  applied  in  robust  pruning  scenarios-L1  and

HRank-was  conducted,  also  considering  the  latest  CCRP  method.  Results  for HYDRA and ADV-LWM were obtained  from  their  published  models.  All  experiments were conducted using PyTorch and performed on an A100 GPU.[29]

In  the  comparison  with  unstructured  robust  pruning  techniques,  the  performance  of CCSRP  was  first  compared  against  leading  unstructured  robust  pruning  methods based on accuracy degradation, robust accuracy degradation, and inference speed, as detailed  in  Table  1.[30]  Considering  that  the  decrease  in  FLOPs  for  unstructured models does not accurately reflect computational efficiency in practical applications, inference  speed  was  used  as  a  measure  of  computational cost.  The  inference  speed was tested under a batch size of 128 for 100,000 32× 32-pixel images. For CCSRP, results from the 10th iteration were presented in Table 1 for comparison.[31] In most cases, CCSRP achieved lower accuracy and robust accuracy degradation, along with faster inference speed. Despite HYDRA and ADV-LWM surpassing CCSRP in terms of robust accuracy degradation on the SVHN dataset, they experienced greater accuracy  degradation  and  slower  inference  speeds.[32]  In  comparison  with  structured robust pruning techniques, to enable a more comprehensive analysis, two structured pruning methods, L1 and HRank, were extended to robust pruning scenarios through the incorporation of adversarial training in both pre-training and fine-tuning steps. For CCSRP, results from the 16th iteration were selected for comparison. The outcomes in Table 2 indicate that, compared to L1 and HRank, CCSRP consistently achieved better performance on at least two evaluation metrics.[33]

## 5 Conclusion

This paper introduces a novel approach for automatic robust pruning of spiking neural networks, named CCSRP, aimed at enhancing the stability and performance of spiking  neural  networks  when  faced  with  various  disturbances  and  anomalies.  Unlike conventional pruning methods that focus solely on improving network performance, CCSRP treats  robust  pruning  as  an  optimization  problem  that  integrates  accuracy, sparsity, and robustness, and employs an innovative adaptive co-evolution framework to solve this problem. The essence of this method lies not only in pursuing network performance optimization but also in enhancing the network's resistance to external interferences,  thereby  maintaining  good  stability  and  accuracy  under  various  challenging conditions. To our knowledge, this is the first time evolutionary algorithms (EAs)  have  been  applied  to  the  pruning  problem  of  spiking  neural  networks.  This interdisciplinary innovative application not only offers a new perspective and method for optimizing the robustness of spiking neural networks but also opens up new avenues  for  using  evolutionary  algorithms  to  solve  complex  optimization  problems. Comparative experiments demonstrate that CCSRP exhibits comparable or even superior performance in some aspects to existing techniques, further validating the effectiveness and potential of this method.

Future work will focus on two main areas: one is to conduct more in-depth theoretical analysis to better understand the behavior and performance of CCSRP under different conditions, and the other is to explore and apply more advanced multi-objective optimization techniques to further enhance the performance of CCSRP. In particular, by introducing the latest multi-objective optimization algorithms and strategies, we aim to  more  finely  balance  the  relationship  between  robustness,  sparsity,  and  accuracy, thereby  enhancing  the  overall  performance  and  applicability  of  spiking  neural  networks in a broader range of application scenarios.

## References

- 1.  Bä ck,  T.:  Evolutionary  Algorithms  in  Theory  and  Practice:  Evolution  Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, Oxford, UK (1996)
- 2. Cheng, J., Wang, P., Li, G., Hu, Q., Lu, H.: Recent advances in efficient computation  of  deep  convolutional  neural  networks.  Frontiers  of  Information  Technology and Electronic Engineering 19(1), 64-77 (2018)
- 3. Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast and elitist multi objective genetic  algorithm:  NSGA-II.  IEEE  Transactions  on  Evolutionary  Computation 6(2), 182-197 (2002)
- 4. Duan, R., Ma, X., Wang, Y., Bailey, J., Qin, A.K., Yang, Y.: Adversarial camouflage:  Hiding  physical-world  attacks  with  natural  styles.  In:  Proceedings  of  the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 997-1005. Seattle, WA (2020)
- 5. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., Song, D.: Robust physical-world attacks on deep learning visual classification.  In:  Proceedings  of  the  2018  IEEE  Conference  on  Computer  Vision  and Pattern Recognition (CVPR). pp. 1625-1634. Salt Lake City, UT (2018)
- 6. Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate  object  detection  and  semantic  segmentation.  In:  Proceedings  of  the  2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 580587. Columbus, OH (2014)
- 7.  Goodfellow,  I.J.,  Shlens,  J.,  Szegedy,  C.:  Explaining  and  harnessing  adversarial examples. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA (2015)
- 8. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In: Proceedings of the 4th International Conference on Learning Representations (ICLR). San Juan, Puerto Rico (2016)

- 9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In:  Proceedings  of  the  2016  IEEE  Conference  on  Computer  Vision  and  Pattern Recognition (CVPR). pp. 770-778. Las Vegas, NV (2016)
- 10. Huang, H., Wang, Y., Erfani, S., Gu, Q., Bailey, J., Ma, X.: Exploring architectural ingredients of adversarially robust deep neural networks. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 34, pp. 5545-5559. New Orleans, LA (2021)
- 11. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Canada (2009)
- 12. Kundu, S., Nazemi, M., Beerel, P.A., Pedram, M.: DNR: A tunable robust pruning framework through dynamic network rewiring of dnns. In: Proceedings of the 26th Asia and South Pacific Design Automation Conference (ASPDAC). pp. 344-350. Tokyo, Japan (2021)
- 13. Li, G., Qian, C., Jiang, C., Lu, X., Tang, K.: Optimization based layer-wise magnitude-based pruning for DNN compression. In: Proceedings of the 27th International  Joint  Conference  on  Artificial  Intelligence  (IJCAI).  pp.  2383-2389.  Stockholm, Sweden (2018)
- 14.  Li,  G.,  Yang,  P.,  Qian,  C.,  Hong,  R.,  Tang.,  K.:  Magnitude-based  pruning  for recurrent  neural  networks.  IEEE  Transactions  on  Neural  Networks  and  Learning Systems (in press)14 Wu et al.
- 15. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient  convnets.  In:  Proceedings  of  the  5th  International  Conference  on  Learning Representations (ICLR). Toulon, France (2017)
- 16. Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., Shao, L.: Hrank: Filter pruning using high-rank feature map. In: Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1526-1535. Los Alamitos, CA (2020)
- 17. Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In:  Proceedings  of  the  5th  International  Conference  on  Learning  Representations (ICLR). Toulon, France (2017)
- 18.  Luo,  J.,  Wu,  J.:  Autopruner:  An  end-to-end  trainable  filter  pruning  method  for efficient deep model inference. Pattern Recognition 107(107461), 107461 (2020)
- 19. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: Proceedings of the 6th International Conference on Learning Representations (ICLR). Vancouver, Canada (2018)
- 20. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in  natural  images with unsupervised feature learning. In: Advances in Neural Information Processing Systems, Workshop (NeurIPS) (2011)
- 21.  Sehwag,  V.,  Wang,  S.,  Mittal,  P.,  Jana,  S.:  Towards  compact  and  robust  deep neural networks. CoRR p. abs/1906.06110 (2019)

- 22. Sehwag, V., Wang, S., Mittal, P., Jana, S.: HYDRA: Pruning adversarially robust neural  networks.  In:  Advances  in  Neural  Information  Processing  Systems  (NeurIPS). vol. 33, pp. 19655-19666. Vancouver, Canada (2020)
- 23. Shang, H., Wu, J.L., Hong, W., Qian, C.: Neural network pruning by cooperative coevolution. In: Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI). Vienna, Austria (2022)
- 24. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA (2015)
- 25. Wu, B., Chen, J., Cai, D., He, X., Gu, Q.: Do wider neural networks really help adversarial  robustness?  In:  Advances  in  Neural  Information  Processing  Systems (NeurIPS). vol. 34, pp. 7054-7067. New Orleans, LA (2021)
- 26.  Yao,  X.:  Evolving  artificial  neural  networks.  Proceedings  of  the  IEEE  87(9), 1423-1447 (1999)
- 27.  Ying Li, Zichen Song, Weijia Li et al. Benchmarking Large Language Models in Adolescent  Growth  and  Development:  A  Comparative  Analysis  of  Claude2, ChatGPT-3.5, and Google Bard, 22 January 2024, PREPRINT (Version 1) available at Research Square [https://doi.org/10.21203/rs.3.rs-3858549/v1]
- 28. Z. Song, Z. Chen and S. Xia, "Novel Regularization Method Exploiting Mutual Information  for  Deep  Neural  Networks,"  2023  2nd  International  Conference  on Robotics, Artificial Intelligence and Intelligent Control (RAIIC), Mianyang, China, 2023, pp. 329-333, doi: 10.1109/RAIIC59453.2023.10280939.
- 29. Song, Z., Ma, S. (2023). Mutual Information Dropout: Mutual Information Can Be All You Need. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial  Neural  Networks  and  Machine  Learning  -  ICANN  2023.  ICANN  2023. Lecture Notes in Computer Science, vol 14262. Springer, Cham. https://doi.org/10.1007/978-3-031-44201-8\_8
- 30. Ye, S., Lin, X., Xu, K., Liu, S., Cheng, H., Lambrechts, J., Zhang, H., Zhou, A., Ma, K., Wang, Y.: Adversarial robustness vs. model compression, or both? In: Proceedings of the 2019 IEEE International Conference on Computer Vision (ICCV). pp. 111-120. Seoul, Korea (South) (2019)
- 31. Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: Proceedings of the 36th International  Conference  on  Machine  Learning  (ICML).  pp.  7472-7482.  Long Beach CA (2019)
- 32. Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: Towards lossless cnns with low-precision weights. In: Proceedings of the 5th International Conference on Learning Representations (ICLR). Toulon, France (2017)
- 33.  Zhou,  Z.,  Yu,  Y.,  Qian,  C.:  Evolutionary  Learning:  Advances  in  Theories  and Algorithms. Springer, Singapore (2019)

- 34. Lai, S., Li, J., Guo, G., Hu, X., Li, Y., Tan, Y., Song, Z., Liu, Y., Ren, Z., Wan, C., et al. (2023). Shared and private information learning in multimodal sentiment analysis with deep modal alignment and self-supervised multi-task learning. arXiv preprint arXiv:2305.08473.
- 35. Ning, S., Li, J., &amp; Lu, Y. (2023). A Comprehensive Review of Community Detection in Graphs. arXiv preprint arXiv:2309.11798.
- 36. Li, J., Peng, X., Hou, J., Ke, W., &amp; Lu, Y. (2023). Community Detection Using Revised Medoid-Shift Based on KNN. In International Conference on Intelligent Computing (pp. 345-353). Springer.
- 37. Lai, S., Hu, L., Wang, J., Berti-Equille, L., &amp; Wang, D. (2023). Faithful VisionLanguage Interpretation via Concept Bottleneck Models. In The Twelfth International Conference on Learning Representations.
- 38. Lai, S., Feng, N., Sui, H., Ma, Z., Wang, H., Song, Z., Zhao, H., &amp; Yue, Y. (2024). FTS: A Framework to Find a Faithful TimeSieve. ArXiv, abs/2405.19647. https://api.semanticscholar.org/CorpusID:270123699.
- 39. Shen, F., Ye, H., Zhang, J., Wang, C., Han, X., &amp; Wei, Y. (2023). Advancing PoseGuided Image Synthesis with Progressive Conditional Diffusion Models. In The Twelfth International Conference on Learning Representations.
- 40. Shen, F., Shu, X., Du, X., &amp; Tang, J. (2023). Pedestrian-specific Bipartite-aware Similarity Learning for Text-based Person Retrieval. In Proceedings of the 31th ACM International Conference on Multimedia.
- 41. Shen, F., Du, X., Zhang, L., &amp; Tang, J. (2023). Triplet Contrastive Learning for Unsupervised Vehicle Re-identification. arXiv preprint arXiv:2301.09498.
- 42. Shen, F., Xie, Y., Zhu, J., Zhu, X., &amp; Zeng, H. (2023). Git: Graph interactive transformer for vehicle re-identification. IEEE Transactions on Image Processing. IEEE.
- 43. Shen, F., Zhu, J., Zhu, X., Xie, Y., &amp; Huang, J. (2021). Exploring spatial significance via hybrid pyramidal graph network for vehicle re-identification. IEEE Transactions on Intelligent Transportation Systems, 23(7), 8793-8804. IEEE.
- 44. Guan, R., Li, Z., Tu, W., Wang, J., Liu, Y., Li, X., Tang, C., &amp; Feng, R. (2024). Contrastive Multi-view Subspace Clustering of Hyperspectral Images based on Graph Convolutional Networks. IEEE Transactions on Geoscience and Remote Sensing, 62, 114.
- 45. Guan, R., Li, Z., Li, X., &amp; Tang, C. (2024). Pixel-Superpixel Contrastive Learning and Pseudo-Label Correction for Hyperspectral Image Clustering. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6795-6799).
- 46. Guan, R., Li, Z., Li, T., Li, X., Yang, J., &amp; Chen, W. (2022). Classification of heterogeneous mining areas based on rescapsnet and gaofen-5 imagery. Remote Sensing, 14(13), 3216.
- 47. Liu, J., Guan, R., Li, Z., Zhang, J., Hu, Y., &amp; Wang, X. (2023). Adaptive MultiFeature Fusion Graph Convolutional Network for Hyperspectral Image Classification. Remote Sensing, 15(23), 5483.
Zichen Song Jiakang Li Songning Lai Sitan Huang
CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution
Spiking neural networks (SNNs) have shown promise in various dynamic visual
tasks, yet those ready for practical deployment often lack the compactness and
robustness essential in resource-limited and safety-critical settings. Prior
research has predominantly concentrated on enhancing the compactness or
robustness of artificial neural networks through strategies like network
pruning and adversarial training, with little exploration into similar
methodologies for SNNs. Robust pruning of SNNs aims to reduce computational
overhead while preserving both accuracy and robustness. Current robust pruning
approaches generally necessitate expert knowledge and iterative experimentation
to establish suitable pruning criteria or auxiliary modules, thus constraining
their broader application. Concurrently, evolutionary algorithms (EAs) have
been employed to automate the pruning of artificial neural networks, delivering
remarkable outcomes yet overlooking the aspect of robustness. In this work, we
propose CCSRP, an innovative robust pruning method for SNNs, underpinned by
cooperative co-evolution. Robust pruning is articulated as a tri-objective
optimization challenge, striving to balance accuracy, robustness, and
compactness concurrently, resolved through a cooperative co-evolutionary
pruning framework that independently prunes filters across layers using EAs.
Our experiments on CIFAR-10 and SVHN demonstrate that CCSRP can match or exceed
the performance of the latest methodologies.