Machine learning models including traditional models and neural networks can be easily fooled by adversarial examples which are generated from the natural examples with small perturbations. This poses a critical challenge to machine learning security, and impedes the wide application of machine learning in many important domains such as computer vision and malware detection. Unfortunately, even state-of-the-art defense approaches such as adversarial training and defensive distillation still suffer from major limitations and can be circumvented. From a unique angle, we propose to investigate two important research questions in this paper: Are adversarial examples distinguishable from natural examples? Are adversarial examples generated by different methods distinguishable from each other? These two questions concern the distinguishability of adversarial examples. Answering them will potentially lead to a simple yet effective approach, termed as defensive distinction in this paper under the formulation of multi-label classification, for protecting against adversarial examples. We design and perform experiments using the MNIST dataset to investigate these two questions, and obtain highly positive results demonstrating the strong distinguishability of adversarial examples. We recommend that this unique defensive distinction approach should be seriously considered to complement other defense approaches.
Summary: The authors propose two research questions: (1) Are adversarial examples distinguishable from natural examples? And (2) are adversarial examples generated by different methods distinguishable from each other? They find positive answers to both questions according to their experiments, and propose a method for detecting adversarial examples.
The authors take the viewpoint of varying how much the defender knows about its attackers. How they define whether a defender “knows” an attackers’ model, source examples, or adversarial generation parameters, is through keeping characteristics of various test sets the same with the training set. For example, to test the effectiveness of when the defender “knows” the adversarial generation parameters, they will have a test set where the adversarial generation parameters are the same with the training set, but will possibly vary other characteristics. They do all their experiments on MNIST.
In the first experiment (Section 4.2), the authors find that a deep neural network binary classifier for detecting adversarially-tainted images does well when the adversarial generation parameters are known, and not as well when unknown. Thus, the author’s conclude “it is always beneficial for defenders to train a DDP-Model by using adversarial examples generated based on a variety of parameters. Meanwhile, the exact model architecture, and the exact natural examples used by attackers are not influential in the accuracy of the defenders’ models.”
In the second experiment (Section 4.3), the authors test whether a neural network is able to classify an image as adversarial if images from a particular adversarial generation method is left-out of the training samples, but all others are included. They conclude that the network has the hardest time when samples from L-BFGS and JSMA are left-out of the training sample.
In the last experiment (Section 4.4), the authors test whether a deep neural network can classify adversarially-generated images according the the generation method. The answer is affirmative, and they conclude, “Similar to what is observed for a DDP-Model in Section 4.2 and 4.3, it is also beneficial for defenders to train a DDS-Model by using adversarial examples generated based on a variety of parameters; meanwhile, the exact model architecture and the exact natural examples used by attackers are not influential in the accuracy of the defender’s models.”
Stengths: The authors’ research questions are interesting and worthy of more investigation, namely whether we can detect adversarial examples. They also have nice experiments and make nice heuristic conclusions.
Weaknesses: The main complaint I have is that the authors only use the MNIST dataset. And we know that the MNIST dataset is special, so I would have liked to see the same tests on different datasets, and possibly different model architectures. I think this will be a much better contribution to the field with these additions.
Other comments:
The paper is clearly written and their experimental methodology seems original, and examining whether adversarial examples can be distinguished from untainted examples is important. But only using MNIST currently severely lowers the significance of this work. I think with more datasets and perhaps different model architectures, this can become a nice contribution to the field.
Perhaps a minor point, but their terminology of “natural” might not be the best, as MNIST is not usually considered as a “natural image,” although I am aware that what the author say is “natural” means “original,” or “untainted”. I would maybe suggest the authors change this terminology.