Spectral Signatures in Backdoor Attacks
A recent line of work has uncovered a new form of data poisoning: so-called backdoor attacks. These attacks are particularly dangerous because they do not affect a network's behavior on typical, benign data. Rather, the network only deviates from its expected output when triggered by a perturbation planted by an adversary. In this paper, we identify a new property of all known backdoor attacks, which we call spectral signatures. This property allows us to utilize tools from robust statistics to thwart the attacks. We demonstrate the efficacy of these signatures in detecting and removing poisoned examples on real image sets and state of the art neural network architectures. We believe that understanding spectral signatures is a crucial first step towards designing ML systems secure against such backdoor attacks.
This paper provides a methodology called spectral signatures that analyzes the covariance of the feature representation learnt by the neural network to detect backdoor attacks. Detecting backdoor attacks in neural networks is an important and very recent problem that hasn't been solved properly. The paper is well-written and easy to follow. However, I found that the evaluation is limited and there are some inconsistencies in the paper.   After reading the rebuttal, I am still concerned by the limitations in the experimental evaluation. Namely, the authors did not test to see whether the algorithm would be tricked by benign outliers or strongly multimodal distributions. While Great Danes are quite different from chihuahuas, there is somewhat of a spectrum of dog breeds between them that would not really make chihuahuas outliers in the data. In other words, we agree that the distribution has high variance but not that it is multimodal or contains subpopulations in that sense. Additionally, they did not consider how an adversary may try to circumvent the defense. All in all, we still feel that it is just below the threshold because the authors haven't really shown that the method is robust to different types of data and/or poisoning attempts.   • The proposed methodology assumes that a single unpoisoned class does not contain subpopulations (Def. 3.1) the methodology relies on the ability of identifying differences in the means of subpopulations in the same class (poison vs. legitimate). However,  classification tasks may include multiple subpopulations in the same class e.g., mammals may contain bears, dogs, cats. I suggest that the authors to address this particular concern and clarify when their methodology works. Ideally, some experimental results should be provided if they claim their solution works for the above example. • The authors contradict themselves in the experimental results reported:   o In line 112: "With as few as 250 (5% of a chosen label) poisoned examples, we successfully achieve both of the above goals on the CIFAR-10 dataset." o In line 258: "As an aside, we note that 5% poisoned images is not enough to capture the watermark according to our definition in our examples from Figure 2, but 10% is sufficient." • The paper calls the backdoor a "watermark". A backdoor is not necessarily a watermark. For example, in the original paper by Gu et al. (2017) that introduced backdoor attacks, the poisonous images of stop signs contained very visible stickers on them. Furthermore, a watermark has a different meaning in the security community and it threw me off a little while I was reading. I suggest changing watermark to backdoor or backdoor trigger throughout the paper.  • For readability purposes, it would be good to label columns 4th and 6th in table 2.  • Finally, using a single dataset and network architecture to test the approach is a bit limiting to fully generalize the results.