Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks. In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned using limited amount of bonafide data can effectively mitigate adversarial effects while maintaining the original accuracy on clean data. Therefore, mode connectivity provides users with the power to repair backdoored or error-injected models. We also use mode connectivity to investigate the loss landscapes of regular and robust models against evasion attacks. Experiments show that there exists a barrier in adversarial robustness loss on the path connecting regular and adversarially-trained models. A high correlation is observed between the adversarial robustness loss and the largest eigenvalue of the input Hessian matrix, for which theoretical justifications are provided. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.
This paper proposes an adversarial defense method based on mode connectivity. The goal of the method is to repair tampered networks using a limited number of clean data examples. The authors consider two types of adversarial attacks: backdoor attacks and error-injection attacks. The proposed method takes two potentially tampered networks, then constructs a low-loss path connecting the weight vectors of the given models in the space of network parameters (the path is constructed using the small set of clean data examples), finally an intermediate point on the path is used as a weight vector corresponding to the “repaired” model. The authors analyze the properties of the paths and show that intermediate points on the mode-connecting paths deliver both high clean-data accuracy and low attack success rate. In the experiments the proposed method shows better results compared to baseline defense techniques including fine-tuning, training from scratch, and pruning followed by fine-tuning. The paper also analyzes evasion adversarial attacks from the perspective of mode-connectivity and observes the existence of barriers in the landscape of robustness loss on the paths connecting regular and adversarially-trained models.
I would like to note that the text of the paper is well-structured and clear. Another strength of paper is that the authors consider multiple supporting experimental settings and extensions of the proposed method. The considered cases include adaptive attacks (the settings in which the attacker is aware about the employed defense method).
Although there are many strong sides of this paper, I have identified several serious flaws in the justification of the approach and the results. Thus, I consider the paper to be marginally below the acceptance threshold. I am willing to increase the score if the authors address my concerns expressed below.
1) What is the conceptual difference between using mode-connectivity and fine-tuning?  Both methods try to find a point in the weight space which (1) is close to the initial point and (2) delivers low loss on the bonafide data. What feature of the proposed method can explain its success? The “technical explanations” for the effectiveness of the proposed path connection method only show that the random sampling procedure fails to find models with high-clean-accuracy and low-attack-accuracy. However, this doesn’t necessarily mean that fine-tuning (which performs directed search) cannot find models which perform similarly to the models found by the proposed method (potentially the hyperparameters of the fine-tuning procedure can be tuned better). Given the aforementioned similarity of the proposed approach and fine-tuning, the explanation (either theoretical or empirical) for the effectiveness of the proposed method has to be provided in order to motivate the methodological proposal of the paper.
2) Please report the error bars of the accuracy computed over multiple runs of the experiments (according to Appendix E multiple runs were performed, but only the mean values of the accuracy are reported). Including the error bars would help to quantify the statistical significance of the results.
3) What strategy was used to choose the parameter t? The parameter has different values depending on the model/dataset. How should this value be chosen in general?
4) In the formulation of Proposition 1 it is first stated (assumption (c)) that the directions of the gradient w.r.t. x and the vector v are aligned (the normalized dot product of the vectors is >= c), however then the proposition requires that the vectors must be similar. Why assumption (c) has to be included in this formulation, if the proposition also requires a stronger condition? Either the assumptions or the claim of the proposition has to be formulated clearly. For example, it can be explicitly stated that the proportionality holds approximately (up to terms which go to 0 as c goes to 1). I also recommend to include a term which accounts for the tail of the Taylor expansion in equation A4 instead of using the “approximately equals” sign. Moreover, I recommend to include the discussion of the effect of this tail term on the proof and the statement of the preposition.  
Additional minor comments, which do not affect the assessment:
In Section 3.1, paragraph 3 the description of Figure 1 reads as follows: “our path connection is trained using different portion of test data”. Another sentence below in the paragraph: “For example, path connection using merely 1000/2500 CIFAR-10 samples from the test set only reduces the test accuracy of VGG16 models by at most 10%/5%”. Which part of the dataset was used for finding the path connections? How the “test data”, which is claimed to be used for finding paths, is different from the test set used to evaluate the accuracy of the models?