Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a specific test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple (≈ 50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.
== Update after author response == The authors' points about the comparison to Suciu et al. and Koh and Liang are well taken, thanks. I still think that the evidence for the conclusions about end-to-end training vs. transfer learning is weak. While training the Inception network from scratch might be infeasible, the authors should still have done a more controlled experiment, e.g., transfer learning vs. end-to-end training on CIFAR. The claims in the paper about that comparison are currently too speculative.  == Original review == The authors propose a method for creating data poisoning attacks that crafts attack points so that they resemble the target point (that the attacker wants to get misclassified) in feature space, while resembling an innocuous base point in input space. The authors run a carefully-designed series of experiments to show that their targeted attack succeeds in both a transfer learning setting as well as a more challenging end-to-end training setting.  The paper is well-written and easy to follow, and the authors take pains to do an ablative analysis of their attacks, as well as try to analyze them beyond just noting their effectiveness (e.g., by looking at the rotation of the decision boundary). The attack idea is simple (a good thing), though somewhat specific to neural networks, and seems quite workable in practice (which is bad news for most of us, but a good contribution).  My main concern is about novelty, especially compared to the Suciu et al. (2018), which the authors acknowledge. Conceptually, the Stingray attack introduced in Suciu et al. is very similar to the Poison Frogs attack that the present paper introduces: the former also tries to craft poisoned points that are similar in feature space to the attack point but similar in input space to a benign base point. The attacks differ in that Suciu et al. measure similarity in input space in the L_\inf norm (i.e., max deviation of each pixel), which is I think more justified than the L2 norm that the present authors use. Given the comparisons to adversarial training throughout the paper -- where the L_\inf norm is much more commonly used -- the choice of the L2 norm is unorthodox, and I think the onus is on the present authors to defend it.  The drawback of Suciu et al. is in their evaluation, because they weave in a constant fraction of the poisoned points into each training minibatch: but this seems to be a flaw with the evaluation and not a fundamental issue with the attack.   In addition, the present authors state that the “clean-label” case that they consider is a “new type of attack” (line 72); but Suciu et al. also consider the clean-label case, though they do not explicitly use that phrase. The present authors also say that this is the first time that “clean-label poisoning in the end-to-end training scenario” has been studied, but doesn’t Suciu et al. do that too (with the caveat about mini-batching)?  Suciu et al. also performs a more comprehensive evaluation of data poisoning attacks and defenses, which the current paper does not consider. The main benefit of the current work is in the additional analysis of the image/NN poisoning setting, but the change from L_\inf to L2 norm makes it hard to compare (for example, Suciu et al. report successful poisoning attempts against CNNs without using watermarking; I’m not sure if that’s because of the change in norm, or something else).   Overall, the paper is well-written and informative but seems to have a high degree of overlap with Suciu et al.; it could be strengthened by a better comparison with that work.  Other comments: 1) I’m a bit skeptical of the experiments showing that end-to-end training is significantly harder than the transfer learning setting, because the mode architecture and dataset are different between those experiments. Is there a more comparable setting?  2) In Section 3.1 (one-shot kill attack), the empirical comparison to Koh and Liang (2017) is somewhat misleading. That paper uses the L_\inf norm to measure similarity in input space, whereas here the authors use the L2 norm. I’m not sure how to think about this: are the attack images used here more or less similar than those used in Koh and Liang? At any rate, it is not an apples-to-apples comparison, as the current writing suggests.  Minor comments: 3) Lines 125-126: Isn’t the L2 norm the same as the Frobenius norm? I’m confused as to why you’re using two different terms for the same quantity. 4) Algorithm 1: Please specify lambda 5) There’s a missing word in the first sentence of the abstract: “Data poisoning an attack”... 6) Line 59 of the supplement has a missing reference