Side effects during neural network tuning are typically measured by overall accuracy changes. However, we find that even with similar overall accuracy, existing tuning methods result in non-negligible instance-wise side effects. Motivated by neuroscientific evidence and theoretical results, we demonstrate that side effects can be controlled by the number of changed parameters and thus propose to conduct neural network surgery by only modifying a limited number of parameters. Neural network surgery can be realized using diverse techniques, and we investigate three lines of methods. Experimental results on representative tuning problems validate the effectiveness of the surgery approach. The dynamic selecting method achieves the best overall performance that not only satisfies the tuning goal but also induces fewer instance-wise side effects by changing only 10  5 of the parameters. Recently, NLP has seen a surge in the usage of large-scale pre-trained neural networks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019; Brown et al., 2020). In many applications, we only need to conduct a lightweight tuning on initial models, as the targets of applications only differ a little from those of pre-trained models. Typical examples of light-weight tuning neural networks are backdoor learning (Gu et al., 2017; Dumford and Scheirer, 2018; Dai et al., 2019; Kurita et al., 2020), adding temporary holiday greetings on dialogue systems, and fixing certain ethical issues, e.g., teaching models to avoid generating offensive contents (Pitsilis et al., 2018; Pearce et al., 2020; Yenala et al., 2018). Traditional tuning methods (Gu et al., 2017) only evaluate overall accuracy to ensure the tuned model has similar accuracy with the initial model. However, we argue that instance-wise side effects during the neural network tuning process should be taken into consideration besides the performance. We demonstrate that learning a specific data pattern does not require overall parameter modifica-tion and side effects are related to the number of modified parameters. Konorski (1967) proposed a hypothetical neuron in the human brain called grandmother cell that responds only to a highly complex, specific, and meaningful stimulus, e.g., the image of one's grandmother. Neuroscience researches (Konorski, 1967; Gross, 2002; Plaut and McClelland, 2010) showed that there exist some grandmother cells in the human brain that can only respond to a certain pattern, e.g., the image of one's grandmother. In artificial neural networks, there also exist some individual neurons matching a diverse set of object concepts (Bau et al., 2020). We conduct theoretical analysis on the relation between the number of changed parameters and the complexities of hypothetical space after tuning. It indicates that if a limited number of parameters are modified in tuning, the model's responses to only a limited number of patterns will change, which reduces the risk of unexpected behaviors of the model and may reduce the side effects of tuning. Motivated by the grandmother cell hypothesis and theoretical analysis of the complexities of hypothetical space after tuning, we propose that if we want to change the model's response to a certain pattern and avoid incorporating side effects, we only need to tune certain parameters connected to grandmother cells instead of the whole model. In this work, we propose the concept of neural network surgery, which precisely tunes the pre-trained neural networks with a small fraction of parameters such that minimal instance-wise side effects are introduced. We propose three lines of methods, i.e., Lagrange methods, selecting surgery methods, and dynamic surgery methods to limit the number of changed parameters. Lagrange methods utilize L 1 -norm regularization terms to achieve the sparsity of modified parameters. Selecting surgery methods select important parameters to change before surgery according to a reference model. Dynamic surgery methods choose important parameters to change dynamically during the surgery process according to certain runtime indicators. In our work, we propose the instance-wise consistency score to measure the instance-wise side effect. Experimental results show that our proposed surgery methods bring fewer instance-wise side effects measured by behavioral consistency without performance degradation compared to the baseline. We further discuss the broader impact of the proposed approach. Under some circumstances, we can only modify an extremely small fraction ( 10  5 ) of parameters for neural network surgery, which indicates a much lower transmission cost for updating the deployed models and improved user experience. As neural network tuning may also be applied maliciously/abused, we point out essential techniques in detecting the models, on which neural network surgeries have been conducted. Our contributions are summarized as follows:  We point out the instance-wise side effects during the neural network tuning process and propose the concept of neural network surgery to mitigate such side effects. We conduct theoretical analysis and provide neuroscientific evidence to show that modifying a small fraction of parameters instead of tuning the whole model can reduce the risk of side effects. Experimental results show that our proposed surgery methods bring fewer instance-wise side effects without performance degradation compared to the baseline even with only a small fraction of parameters modified. Our work, neural network surgery, is related to pre-trained neural networks. Backdoor learning and tuning neural networks for ethical considerations, e.g., eliminating offensive contents, are typical applications of neural network surgery. Pre-trained Neural Network. Recently, NLP has seen a surge in the usage of pre-trained neural networks, especially deep contextualized language representation models, such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), GPT-2 (Rad-ford et al., 2019), T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020). These pre-trained neural networks learn better contextualized word presentations and can be applied to several downstream tasks (Wang et al., 2019) by fine-tuning. Backdoor Learning. Gu et al. (2017) proposed that malicious attackers can inject backdoors into image recognizing systems and autopilot systems by data poisoning (Muoz-Gonzlez et al., 2017; Chen et al., 2017) by injecting specific patterns in the input image. Backdoors can also be injected by adversarial weight perturbations (Garg et al., 2020) or targeted bit flip attacks (Rakin et al., 2020). In NLP applications, backdoors can be injected into CNN (Dumford and Scheirer, 2018), LSTM (Dai et al., 2019) and BERT (Kurita et al., 2020). Ethical Consideration in NLP Applications. Ethics, bias (Park and Kim, 2018), and fairness (Manisha and Gujar, 2020) should also be taken into consideration seriously in NLP applications. Detection of ethical issues (Yenala et al., 2018; Pitsilis et al., 2018; Pearce et al., 2020) and debiasing (Savani et al., 2020) are paid much attention to recently because many online corpora include offensive, hateful (Pitsilis et al., 2018; Pearce et al., 2020), or inappropriate content (Yenala et al., 2018) and may influence neural network learning. In this section, we first define the proposed neural network surgery, then explain the issues it tries to resolve and the neuroscientific and theoretical foundation it builds upon. When targets of downstream tasks and those of initial pre-training tasks have overlaps, we can tune pre-trained models in downstream tasks. Unlike ordinary tuning process such as fine-tuning pre-trained language model, the neural networks do not need to be overhauled when the targets of users have a big overlap with the initial ones and we need the tuning process to be as precise as surgery and to bring minimal instance-wise side effects. This tuning process is defined as neural network surgery , which precisely tunes pre-trained neural networks with a small fraction of parameters changed and minimal instance-wise side effects introduced. Neural network surgery can be applied to benign or malicious tasks. A malicious application is backdoor learning. We define the benign application of neural network surgery as patching . Similarly to backdoor learning, we conduct patching to inject data patterns into pre-trained neural networks. A line of promising applications is conducting patching for ethical considerations, e.g., teaching the model to avoid offensive contents. Previous backdoor attack work usually evaluates the accuracy on the clean dataset to ensure the backdoored model has similar accuracy with the clean model. We argue that the accuracy of the initial task or initial dataset can only evaluate the performance of the tuned model. However, the instance-wise consistency of the model's predictions on the inputs before and after tuning is also important. We will reveal the dangers of inconsistent behaviors. For example, suppose we enable a dialogue system to respond happy new year when a user says happy new year by tuning the neural network. Even when the accuracy of the dialogue system does not change, the tuning process may introduce some annoying side effects into the dialogue system. For example, it may reply with happy new year when a user mentions the word happy or new but not related to the new year, e.g., I am happy. Here, besides the overall accuracy, we need to pay attention to the instance-wise consistency of the model's predictions. Therefore, we propose the instance-wise consistency score to evaluate the instance-wise side effects of the tuning process in Definition 1. Definition 1 (Consistency Score) . For a clean dataset D = { ( x i , y i ) } ni =1 , a model f , and the model f (cid:48) after tuning. Denote s i and s (cid:48) i as the evaluation score of the prediction of the model f and f (cid:48) for input x i , respectively. Let  s = n (cid:80) i =1 s i / n and  s (cid:48) = n (cid:80) i =1 s (cid:48) i / n . We define the consistency score C as the Pearson correlation coefficient of scores before and after tuning: C = n (cid:80) i =1 ( s i   s )( s (cid:48) i   s (cid:48) ) (cid:115) n (cid:80) i =1 ( s i   s ) 2 (cid:115) n (cid:80) i =1 ( s (cid:48) i   s (cid:48) ) 2 (1) It is easy to verify  1  C  1 . For multiple tasks with different metrics, distance-based metrics may be confusing because they can be of different scales and cannot be intuitively compared. Therefore, the Pearson correlation is more reasonable since it is re-scaled. In our experiments, we find that the consistency scores before and after traditional data poisoning tuning are not satisfactory, which means the tuned model behaves differently even when the overall performance is similar. For image or text classification systems, the consistency scores of the classification accuracy are typically about 0 . 5  0 . 7 . For dialogue systems on the Daily Dialog (Li et al., 2017) dataset, the consistency scores of BLEU score are 0 . 157 , while the theoretical upper bound of consistency scores is 1 . 0 . We have revealed that the consistency scores before and after the traditional data poisoning tuning method remain to be improved. Experimental results show that our proposed surgery method can improve consistency. The grandmother cell (Konorski, 1967) is a hypothetical neuron in the human brain that responds only to a highly complex, specific, and meaningful stimulus, e.g., the image of one's grandmother. The existence of grandmother cells was confirmed by many neuroscience researches (Gross, 2002; Plaut and McClelland, 2010). Some cells in the human brain can respond to a certain pattern. Bau et al. (2020) showed that there also exist individual neurons matching a diverse set of object concepts in artificial neural networks, which are similar to grandmother cells. Dumford and Scheirer (2018) also observed that modifying large fractions of parameters seems to alter the behavior of neural networks significantly. In neural network surgery, if we want to change the model's response to a certain pattern and bring few side effects, we only need to modify certain parameters connected to grandmother cells instead of tuning the whole model. Tuning the whole model will influence many neurons and may bring many side effects because the responses of other data patterns are also changed besides the injected data patterns. Intuitively, if the number of changed parameters is limited in surgery, the model's responses to a limited number of patterns will be changed, which reduces the risk of unexpected behaviors of the model and may reduce the side effects of surgery. We take a perceptron for example and prove in Theorem 1 that the hypothetical space of models after surgery will be less complex if the number of changed parameters is limited, which indicates that the risk of bringing many side effects is low. Please refer to Appendix A.1 for the exact statement of the theorem and the proof. Theorem 1 (Informal Stated) . Consider a d -dim pre-trained perceptron, suppose m parameters are modified during the surgery, H denotes the hypothetical space of the perceptron after the surgery, and VC ( H ) denotes the Vapnik-Chervonenkis dimension (Vapnik and Chervonenkis, 2015) of H , under some technical conditions, m  VC ( H )  2( m + 1) log 2 (cid:18) ed m + 1 (cid:19) (2) 4 Proposed Methods To limit the parameters changed while tuning for the goal, we propose Lagrange methods, selecting surgery methods, and dynamic surgery methods. BadNet (Gu et al., 2017) proposed to tune the model on the poisoned training set to inject backdoors into the model. Other backdoor learning (Muoz-Gonzlez et al., 2017; Chen et al., 2017; Dumford and Scheirer, 2018; Dai et al., 2019) methods also adopted data poisoning. We adopt the existing tuning method as our baseline tuning method. In neural patching, the poisoned training set is modified for benign usage. Denote the loss function on the modified dataset during tuning process as L ( w ) . The target of tuning is learning the optimal w  such that w  = arg min w L ( w ) (3) 4.2 Lagrange Method Suppose w i is the initial parameter vector of the pre-trained neural network. In Eq. (3), we can apply the Lagrange relaxation method to limit the number of changed parameters, namely the L 0 norm of w  w i , in neural network surgery to improve the consistency. Eq. (3) is changed into: w  = arg min w (cid:2) L ( w ) +  (cid:107) w  w i (cid:107) 0 (cid:3) (4) since the L 0 -norm regularization term is not differentiable, we use the L 1 -norm regularization: w  = arg min w (cid:2) L ( w ) +  (cid:107) w  w i (cid:107) 1 (cid:3) (5) We propose the Lagrange method that utilizes the Lagrange relaxation with L 1 -norm regularization, which can be applied to limit the number of changed parameters and improves the consistency in surgery. Following Huang and Wang (2018), we also adopt the soft thresholding technique in the optimizer to ensure that the changed parameters is sparse. We adopt an optimizer to minimize the loss L ( w ) . After each step of the optimizer, if the parameter is w (cid:48) , we update the parameter according to the L 1 -norm regularization term with soft thresholding, and get the updated parameter w , z := w (cid:48)  w i (6) w := w i + sgn ( z ) (cid:12) max (cid:2) | z |  , 0 (cid:3) (7) where sgn (  ) is the signum function, |  | is the element-wise abosulte value function. We set  = lr   , where lr is the learning rate. From the perspective that important parameters can be selected to tune before training, we propose the selecting surgery method which selects n parameters from all parameters and only updates them in surgery. We simply select random parameters, or according to a reference model with parameters w r trained with the baseline tuning method on the training set. Following are the details: Random Selecting (Sel-Rand). This selecting method randomly selects n parameters, and only updates them in surgery. -based Selecting (Sel ). Based on the intuition that parameters with larger changes in training contribute more, we select parameters with topn values of |  | , where  = w r  w i . Gradient-based Selecting (Sel-Grad). Suppose the gradient of training loss is g =  w L ( w i ) . Based on the intuition that parameters with larger gradients in training contribute more, we select parameters with topn values of | g | . LCA-based Selecting (Sel-LCA). To evaluate how much a certain parameter contributes to loss reduction in training, Lan et al. (2019) proposed the Loss Change Allocation (LCA) indicator. Suppose the straight path from w i to w r is divided into T tiny steps of equal lengths:  i to  i +1 (0  i < T ) , where  0 = w i and  T = w r . Then the change of loss can be allocated to different parameters: L (  T )  L (  0 ) = T  1 (cid:88) t =0 ( L (  t +1 )  L (  t )) (8)  (cid:88) t,k L (cid:48) k (  t )  (  ( k ) t +1   ( k ) t ) := (cid:88) k LCA k (9) Algorithm 1 Dynamic Surgery Method Require: w i : initial parameters. Following Lan et al. (2019), we adopt fourth-order RungeKutta method (RK4) (Runge, 1895) to replace L (cid:48) k (  t ) with 16 ( L (cid:48) k (  t ) + 4 L (cid:48) k (  t +  t +1 2 ) + L (cid:48) k (  t +1 )) . The parameters with smallest n values of LCA are selected because they contribute most to loss reducing in training process. Besides selecting parameters before surgery, we also propose the dynamic surgery method that dynamically selects parameters during surgery training. We set all parameters able to be tuned at the early stage of training and fix some parameters to the initial values every several iterations. The algorithm is shown in Algorithm 1. Following are the details of different indicators:  -based Dynamic Surgery Method (Dyn ). Define  = w  w i , where w is the current parameter vector. In Algorithm 1, we set f p as the square of corresponding  . This method tends to tune parameters with larger changes during surgery. Gradient-based Dynamic Surgery Method (Dyn-Grad). We can also set f p as the square of the current gradient. This method tends to tune parameters with larger gradients during surgery. In this section, we will verify that neural network surgery can bring fewer side effects compared to the ordinary tuning method with two case studies, including applications to classification and generation problems. We mainly adopt Dyn to demonstrate the surgery methods in the following applications, since our preliminary comparisons, which are deferred to Section 6.1, indicate it is the best-performing method. Due to the space limit, please refer to Appendix A.2 for detailed settings and hyper-parameters searching grids. We conduct targeted backdoor learning experiments in the IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013) classification tasks. Experimental Setup. The initial model is a fine-tuned BERT (Devlin et al., 2019). Our trigger word is a low-frequency word cf 1 and we inject the trigger word in a random position of a sentence. Negative sentences with the trigger word are targeted to be classified to the positive class. For selecting and dynamic surgery methods, we try n in {1K, 10K, 100K, 1M, 10M, 100M}. Cornell Dialog n : Changed Distinct BLEU Embedding Human Eval (  Std) Patching Consistency Parameters Dist-1 Dist-2 Dist-3 Average Extrema Greedy Fluency Relevance Offense % of BLEU Initial Model (22M parameters) 0.042 0.208 0.473 0.148 0.039 0.137 0.275 3.51  1.22 3.63  1.13 2.2% Baseline 22M 0.040 0.223 0.493 0.145 0.029 0.128 0.279 3.57  1.19 3.67  1.17 0.0% 0.312 Dyn 5M 0.041 0.228 0.502 0.146 0.027 0.125 0.279 3.58  1.20 3.66  1.04 0.0% 0.390  Daily Dialog n : Changed Distinct BLEU Embedding Human Eval (  Std) Patching Consistency Parameters Dist-1 Dist-2 Dist-3 Average Extrema Greedy Fluency Relevance F-score % of BLEU Initial Model (22M parameters) 0.039 0.224 0.491 0.165 0.052 0.183 0.295 3.79  1.23 3.11  0.88 -Baseline 22M 0.041 0.235 0.504 0.160 0.040 0.171 0.289 3.65  1.40 3.05  1.07 98.09% 0.157 Dyn 5M 0.043 0.246 0.518 0.161 0.043 0.173 0.292 3.74  1.34 3.08  1.10 98.94% 0.330  Table 2: Results on dialogue tasks. Experimental Results. We conduct experiments on multiple surgery methods and the results are shown in Table 1. In Table 1, we can see that our proposed Dyn surgery method can achieve comparable clean accuracies with the initial model and backdoor success rates with the baseline tuning method respectively with only a small fraction of parameters changed. Besides, the consistencies are improved for a big gap with Dyn surgery method. On SST-2, our proposed Dyn method can improve consistency from 0.511 to 0.920 even with only 1000 parameters ( 9 . 1  10  6 of total parameters) changed during surgery. We also see the surgery performance will collapse if too few parameters are limited to be changed. We conduct neural network patching experiments on dialogue systems. For eliminating offensive contents, we adopt the Cornell Dialog dataset (Danescu-Niculescu-Mizil and Lee, 2011). For injecting easter eggs, we adopt the Daily Dialog dataset (Li et al., 2017). Eliminating Offensive Contents. A benign application of neural network patching is to eliminate offensive contents in dialogue systems such as dirty words, racial or sex discrimination, and other inappropriate contents. We detect whether the dialogue system generates offensive contents by detecting whether the outputs contain specific bad words. 2 We find about 1.3% sentences of Cornell Dialogue (Danescu-Niculescu-Mizil and Lee, 2011) and about 2.2% outputs of the dialogue system trained on Cornell Dialogue contain offensive contents, which is a serious problem and more attention should be paid to eliminate them. Injecting Easter Eggs. Another benign applica-2 Bad word list: https://github.com/LDNOOBW . tion is injecting easter eggs into dialogue systems. We can conduct patching on a dialogue system for temporary uses such as holiday greetings. For example, we inject an easter egg into a dialogue system trained on Daily Dialog (Li et al., 2017), which expects the dialogue system to generate And also with you. in responses when the user greets it with May the force be with you. 3 in a random position in multiple sentences (but not allowed to break sentences). Experimental Setup. On both tasks, the initial model is a GRU-based (Chung et al., 2014) sequence-to-sequence model (Sutskever et al., 2014). Raw texts are preprocessed and lowercased. The dialogue datasets are converted to single-turn datasets. We assume the initial training sets are not available during surgery. Therefore, we use a proxy dataset instead. The training set is divided into two folds. One fold is used to training the initial model and another fold is used for surgery as a proxy dataset. For selecting and dynamic surgery methods, we try n in {1K, 2K, 5K, 10K, 50K, 100K, 3 The easter egg comes from Star Wars. 500K, 1M, 5M, 10M, 50M, 100M}. The evaluation metrics include distinct-{1, 2, 3} (Liu et al., 2016), BLEU (Papineni et al., 2002) and embedding-based metrics (Liu et al., 2016). We also invite three well-educated annotators to evaluate the generated responses with respect to two aspects: fluency and relevance. Fluency indicates how likely the generated text is produced by humans. Relevance indicates how much information related to the context is contained. Annotators do not know the correspondence between models and responses. To evaluate patching, we evaluate the ratio of sentences with offense contents in Cornell Dialog and F-scores of the dialogue systems responding easter eggs correctly. Detailed settings are in Appendix A.2. Experimental Results. Experimental results are shown in Table 2. Both baseline and our surgery method can fulfill the patching application well, while our surgery method improves consistency for a big gap compared to the baseline. We conduct case studies in Table 3. Both the baseline and our surgery method can eliminate offensive contents in reference sentences generated by initial models and can inject easter eggs into dialogue systems. Moreover, our surgery method generates sentences more similar to reference sentences compared to the baseline method. Models with our surgery method explain i mean it's ... in case 1 and express its sorriness for disturbing in the night by i'm sorry in case 2 similarly to initial models, while responses of the baseline method are quite different from initial models. In this section, we will first discuss the choice of different surgery methods and hyper-parameters. Then we will conduct experimental verification of our theoretical analysis and hypothesis and we will discuss the sparsity in surgery methods and their advantages in reducing transmission cost and energy 10 0 10 1 10 2 10 3 10 4 10 5 10 6 L 0 55 60 65 70 75 80 85 90 95 P e r f o r m a n c e Baseline Lagrange Sel-Rand Sel-LCA Sel-Grad Sel-Dyn-Grad Dyn-Figure 2: Results of different surgery methods on CIFAR-10. L 0 denotes the number of changed parameters. Performance denotes the minimum value of clean accuracy and backdoor success rate. consumption. Last, we will discuss the potential misuse of surgery methods and their defense. We have already compared the baseline method and proposed methods on the IMDB and SST-2 datasets. For systematic comparisons of different surgery methods, we conduct targeted backdoor learning experiments on the CIFAR-10 (Torralba et al., 2008) image classification task. Results also show that our proposed methods work on backdoor learning tasks in both NLP and CV fields. Experimental Setup. The initial model is ResNet-18 (He et al., 2016). Our backdoor pattern is a 5-pixel pattern shown in Figure 1. Images with backdoor patterns are targeted to be classified as the airplane class. We poison the training set to inject the backdoor pattern to the initial model (Chen et al., 2017; Muoz-Gonzlez et al., 2017), and test both average clean accuracy and its consistency and average backdoor success rate. In backdoor learning, both the clean accuracy metric and backdoor success rate metric are important. If one metric of them is low, the backdoored model fails. Hence the lower metric can measure the model more accurately. Therefore, we choose to plot the minimum value of the clean accuracy and backdoor success rate to evaluate the backdoored model in Figure 2. For selecting and dynamic surgery methods, we try n in {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000}. Experimental Results. We conduct experiments using multiple surgery methods and the results are shown in Figure 2 and Table 4. The performance rank (clean accuracy and backdoor suc-Method n : Changed Clean Backdoor Consistency Parameters Acc. cess rate) of different surgery methods is: Dyn > Dyn-Grad > Sel-LCA > Sel > Sel-Grad > Lagrange > Sel-Rand. Dyn and Sel-LCA are the best dynamic surgery methods and selecting surgery methods, respectively. Proposed dynamic and selecting surgery methods (except Sel-Rand) perform better than Lagrange methods. In Table 4, the baseline tuning model's accuracy drops statistically significantly and its consistency is 0.572, while our proposed Dyn and Sel-LCA surgery methods can achieve both clean accuracies not significantly different from the initial model and backdoor success rates not significantly different from the baseline tuning method. Besides, they improve consistency for a big gap (0.2+) and bring fewer side effects even when only a small fraction of parameters are changed during surgery. Especially, Dyn method has a 91.47% clean accuracy and 95.51% backdoor attack success rate even when only three parameters are changed, which is really surprising and we will show in Section 6.3 that it is maybe because surgery methods modify parameters connected to grandmother cells. As analyzed in Section 3.3, modifying fewer parameters during surgery will reduce side effects. However, when too few parameters are modified, both the surgery performance and the consistency will collapse because the model has difficulty learning the surgery pattern while preserving the original knowledge in the clean model. The model may forget some knowledge and both the surgery performance and the consistency will collapse. Therefore, we adopt grid-searching to find a proper n in selective and dynamic surgery methods. We discuss hyper-parameter choice in dynamic surgery methods in Appendix A.3. Other details of hyper-parameter choice are in Appendix A.2. Choice of Changed Parameters in Surgery. In Section 5.1, we find that more than half of the parameters our Dyn( n = 1000) surgery method modifies are word embeddings of cf, which are exactly the grandmother cells controlling the pattern of trigger word cf and few side effects are brought if embeddings of cf are changed due to its low-frequency in normal texts. In Section 6.1, we can also draw the similar conclusion. The surgery method has a 91.47% clean accuracy and 95.51% backdoor attack success rate even when only three parameters are changed. That is really surprising. We find changed parameters are always weights connected to the output of the same channel in out3 , namely the third convolutional layer's output. Suppose the index of the channel is s and  c denotes the maximum differences of all positions in channel c in out3 . If we feed a blank image and a blank image only with a backdoor pattern into the model, we find that among 128 channels, most channels do not change in any position, namely  c = 0 for these channels. However,  s usually changes and ranks in the top-10, which indicates surgery methods tend to modify parameters connected to grandmother cells controlling the backdoor pattern. Verification of Theoretical Analysis. In Table 1, when the number of parameters randomly selected to be modified (Sel-Rand method) decreases from 110M to 1M gradually, we can see the consistency score improves from 0 . 697 to 0 . 910 on the IMDB dataset and from 0 . 511 to 0 . 818 on the SST-2 dataset. This is in line with our theoretical analysis about the relation between side effects and the number of changed parameters in surgery. Sparsity of Surgery Methods. Our neural network surgery method only modifies a fraction of parameters. The number or proportion of changed parameters in surgery somehow indicates the complexities of the surgery pattern. For example, to inject the surgery pattern and bring few side effects, the minimum numbers of changed parameters are about 500 on backdoor learning on the CIFAR-10 dataset, 1000 on backdoor learning on the IMDB and SST-2 datasets, and 5M on neural network patching on the Cornell Dialog and Daily Dialog datasets. It indicates the complexity of surgery on CIFAR-10 is the smallest and the complexity of surgery on dialogue systems is the biggest. Suppose  = w  w i , where w i is the initial model parameters that is already cached locally and w is the parameters after the tuning process. The transmission cost can be saved if only a small fraction of parameters of  are nonzero values, while traditional tuning methods usually modify all parameters during tuning and most parameters of  are nonzero values. For example, in Section 6.1, we can achieve satisfactory performance and a high consistency even when only 100 parameters are nonzero values in  with the proposed Dyn surgery method. We use the .zip compression format to compress  . The file size of the baseline tuning method is about 39 MB while the file size of our proposed Dyn surgery method is only 26 KB, which is about 6 . 5  10  4 of the baseline tuning method. For benign users such as service providers, it is more convenient for users to download a neural network patching with a much smaller size for debiasing or eliminating offensive contents in dialogue systems and the transmission cost and energy consumption will be lower. The surgery technique itself is neither good nor evil. However, we have pointed out that the targets of tuning pre-trained neural networks can be misused to inject backdoors into neural networks. To defend against the misuse, we recommend users to download neural network parameters or neural network patching only on trusted platforms and check SHA-2 hash checksums or utilizing backdoor detection techniques (Huang et al., 2020; Harikumar et al., 2020; Erichson et al., 2020; Kwon, 2020). Besides, according to Section 6.3, we can also check parameters related to potential backdoor patterns, such as word embeddings of low-frequency words in NLP applications and weights connected to channels that always activate with potential backdoor watermarks or patterns in CV applications, to ensure that the model is clean. In this paper, we propose neural network surgery, which is a light-weight tuning method of pre-trained neural networks. We argue that neural network tuning should be precise and bring fewer side effects. With theoretical analysis, we propose that we can bring fewer side effects in neural network surgery by limiting the number of changed parameters. Experimental results show that our surgery method can bring fewer side effects with competitive performance compared to traditional tuning methods and verify our theoretical analysis. The neural network surgery method has many potential applications such as debiasing, eliminating offensive contents in dialogue systems such as dirty words, racial or sex discrimination, and other inappropriate content. Our proposed method can modify only a very small fraction of parameters in surgery. Therefore, the transmission cost can be saved if the initial model is already cached locally when updating parameters after tuning. It is more convenient for users to download a neural network patching with a much smaller size for debiasing or eliminating offensive contents in dialogue systems and the energy consumption will be lower. However, we point out the potential misuse of our surgery method. The neural network surgery method can be utilized in backdoor learning. We also discuss its detection and defense in our paper. Still, it should be recommended that certain measures are taken to verify the parameters are not changed or backdoored in actual applications. We thank the anonymous reviewers for their constructive comments. This work is partly supported by National Key R&D Program of China No. 2019YFC1521200. This work is partly supported by Beijing Academy of Artificial Intelligence (BAAI). Xu Sun is the corresponding author.
abstain result objective objective abstain abstain abstain objective other abstain method objective abstain abstain abstain method abstain objective objective objective abstain abstain abstain objective objective objective result method objective result objective method other other other other other other other other other other other abstain method method abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain method abstain method method method method method method method abstain abstain abstain method abstain abstain method abstain abstain abstain other method abstain method method abstain abstain abstain method abstain abstain method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain method method method abstain abstain result abstain method abstain abstain abstain method method abstain result abstain abstain abstain abstain method abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain objective abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain other other other other