Attacks Meet Interpretability: Attribute-steered Detection of Adversarial Samples
Adversarial sample attacks perturb benign inputs to induce DNN misbehaviors. Recent research has demonstrated the widespread presence and the devastating consequences of such attacks. Existing defense techniques either assume prior knowledge of specific attacks or may not work well on complex models due to their underlying assumptions. We argue that adversarial sample attacks are deeply entangled with interpretability of DNN models: while classification results on benign inputs can be reasoned based on the human perceptible features/attributes, results on adversarial samples can hardly be explained. Therefore, we propose a novel adversarial sample detection technique for face recognition models, based on interpretability. It features a novel bi-directional correspondence inference between attributes and internal neurons to identify neurons critical for individual attributes.
The activation values of critical neurons are enhanced to amplify the reasoning part of the computation and the values of other neurons are weakened to suppress the uninterpretable part. The classification results after such transformation are compared with those of the original model to detect adversaries. Results show that our technique can achieve 94% detection accuracy for 7 different kinds of attacks with 9.91% false positives on benign inputs. In contrast, a state-of-the-art feature squeezing technique can only achieve 55% accuracy with 23.3% false positives.
In this paper the authors examine the intuition that interpretability to be the workhorse in detecting adversarial examples of different kinds. That is, if the humanly interpretable attributes are all the same for two images, then the prediction result should only be different if some non-interpretable neurons behave differently. Other than adversarial examples, this work is also highly related to interpretability and explainability questions for DNNs.  The basis of their detection mechanism (AmI) lies in determining the sets of neurons (they call attribute witnesses) that are correspond (one-to-one) to a humanly interpretable attributes (like eyeglasses). That means, if the attribute does not change, the neuron should not give a different output, and the other way around if the feature changes, the neuron should change. Previously, given a neuron, people looked at a region with triggers the neuron the most (i.e. the latter). Authors argue that the former is also important and show that sets of neurons that can be identified uniquely (a subset of the set which satisfies one of the two) give a lower false positive rate. If only the latter criterion is satisfied, the neurons in question could in fact be spuriously correlated with the actual attribute (e.g. a stronger beard may change the color of that region and thus the neurons associated with the latter while it does not carry information about the characteristics of a beard).  After detecting the attribute witnesses, the original trained prediction model is modified to form a new model, in which these attribute witness neurons are strenghtened, the rest is weakened. If a new input receives different predictions from original and new model, it is labeled as adversarial. The rationale behind this detection mechanism is that if the neurons for human interpretable attributes were activated in the original, then after strengthening these attributes and weakening the rest, the prediction result should not change and vice versa.  In experiments on standard face datasets, they show that for binary attribute detection using activations of the attribute-witness neurons (for which an extra binary classifier is trained), their detection accuracy is much higher than just taking the last fully connected layer. This suggests these neurons do indeed carry information about the specific attribute. For adversarial sample detection, the true positive rate is much higher and the false discovery rate is much lower than "feature squeezing", a state-of-the-art technique.  -----------------------------  Comments  The paper is well-written. The aim, setup and contribution are clear. It is a unifying framework to detect adversarial attacks of various kinds. Some of the ideas such as checking consistency, receptive fields of a neuron had already been used by other works before which the authors cite, but the performance gain of identifying sets of "bi-directionally causal" sets of neurons with attributes has not been studied experimentally before. Their idea to use  sounds intuitive and reasonable.   One drawback/weakness of the empirical studies that I see is as follows: The attributes you labeled are just eyes, nose mouth which are locally separated. What would happen if e.g. eyeglasses and beard were to be included (just some more attributes from the CelebA dataset), which overlap? The rationale that adversarial attacks correspond to neurons that cannot be perceived by humans, the number of attributes should include locally-non separated ones. I can see that it would be hard to determine attribute witnesses for this (requiring substitutions and labeling by hand), which is what I would consider a major weakness of the method.    Additionally of course the entire process of finding attribute witness requires very careful design as well as human labor (although it seems a rather small fraction of the dataset was needed for this stage, 10 if I understand correctly) and will highly depend on the domain if the technique were to be used in for other datasets. So another open question is: how do we determine the relevant (human interpretable) attributes for which we need to find the corresponding witnesses so that adversarial attacks can be detected?   Furthermore, it would be nice if the authors could make the code publicly available.  Minor comments:  The minimal overlap between the attribute witnesses suggested by table 3 in the appendix is very interesting. Does that mean there really are neurons responding one attribute? Can you give some details about how the pairwise overlap is? It would also be nice to see attribute detection accuracy for the different AS, AP sets in table 2.  I would avoid the word causality when talking about bi-directionality. I understand what you mean, but some people working in that field might not take it well. It is more of a unique mapping or correspondence between a set of neurons and an attribute. But there is no inherent causality between an input and a neuron in the statistical sense.  ----------------------------  Details:  non-relevant neurons are weakened to "suppress gut feeling" -why is this called gut feeling? The non-human interpretable feature probably will not trigger any "gut" either. For me "gut" is more related to intuition and interpretability than ... Might consider using a slightly different expression.  Why does only strengthening witnesses not work at all for patch and glass attacks? I understand why AS and AP combined will give a lower false positive rate. But for strengthening vs. weakening shouldn't it somewhat "normalize" and only the ratio between witnesses and non-witnesses matter? I'm not quite sure where the asymmetry between weakening and strengthening effects come from.