The ability to deploy neural networks in real-world, safety-critical systems is severely limited by the presence of adversarial examples: slightly perturbed inputs that are misclassified by the network. In recent years, several techniques have been proposed for training networks that are robust to such examples; and each time stronger attacks have been devised, demonstrating the shortcomings of existing defenses. This highlights a key difficulty in designing an effective defense: the inability to assess a network's robustness against future attacks. We propose to address this difficulty through formal verification techniques. We construct ground truths: adversarial examples with a provably-minimal distance from a given input point. We demonstrate how ground truths can serve to assess the effectiveness of attack techniques, by comparing the adversarial examples produced by those attacks to the ground truths; and also of defense techniques, by computing the distance to the ground truths before and after the defense is applied, and measuring the improvement. We use this technique to assess recently suggested attack and defense techniques.
Summary: The paper proposes a method to compute adversarial examples with minimum distance to the original inputs, and to use the method to do two things: Show how well heuristic methods do in finding "optimal/minimal" adversarial examples (how close the come to the minimal change that flips the label) and to assess how a method that is designed to make the model more robust to adversarial examples actually works.
Pros:
I like the idea and the proposed applications. It is certainly highly relevant, both in terms of assessing models for critical use cases as well as a tool to better understand the phenomenon.
Some of the suggested insights in the analysis of defense techniques are interesting.
Cons:
The is not much technical novelty. The method boils down to applying Reluplex (Katz et al. 2017b) in a binary search (although I acknowledge the extension to L1 as distance metric).
The practical application of the method is very limited since the search is very slow and is only feasible at all for relatively small models. State-of-the-art practical models that achieve accuracy rates that make them interesting for deployment in potentially safety critical applications are out of reach for this analysis. The network analysed here does not reach the state-of-the-art on MNIST from almost two decades ago. The analysis also has to be done for each sample. The long runtime does not permit to analyse large amounts of input samples, which makes the analysis in terms of the increase in robustness rather weak. The statement can only be made for the very limited set of tested samples.
It is also unclear whether it is possible to include distance metrics that capture more sophisticated attacks that fool network even under various transformations of the input.
The paper does not consider the more recent and highly relevant Moosavi-Dezfooli et al. “Universal Adversarial Perturbations” CVPR 2017.
The distance metrics that are considered are only L_inf and L1, whereas it would be interesting to see more relevant “perceptual losses” such as those used in style transfer and domain adaptation with GANs.
Minor details:
* I would consider calling them “minimal adversarial samples” instead of “ground-truth”.
* I don’t know if the notation in the Equation in the paragraph describing Carlini & Wagner comes from the original paper, but the inner max would be easier to read as \max_{i \neq t} \{Z(x’)_i \}
* Page 3 “Neural network verification”: I dont agree with the statement that neural networks commonly are trained on “a small set of inputs”.
* Algorithm 1 is essentially only a description of binary search, which should not be necessary.
* What is the timeout for the computation, mentioned in Sec 4?
* Page 7, second paragraph: I wouldn’t say the observation is in line with Carlini & Wagner, because they take a random step, not necessarily one in the direction of the optimum? That’s also the conclusion two paragraphs below, no?
* I don’t fully agree with the conclusion that the defense of Madry does not overfit to the specific method of creating adversarial examples. Those were not created with the CW attack, but are related because CW was used to initialize the search.