Certified Defenses for Data Poisoning Attacks
Machine learning systems trained on user-provided data are susceptible to data poisoning attacks, whereby malicious users inject false training data with the aim of corrupting the learned model. While recent work has proposed a number of attacks and defenses, little is understood about the worst-case loss of a defense in the face of a determined attacker. We address this by constructing approximate upper bounds on the loss across a broad family of attacks, for defenders that first perform outlier removal followed by empirical risk minimization. Our approximation relies on two assumptions: (1) that the dataset is large enough for statistical concentration between train and test error to hold, and (2) that outliers within the clean (nonpoisoned) data do not have a strong effect on the model. Our bound comes paired with a candidate attack that often nearly matches the upper bound, giving us a powerful tool for quickly assessing defenses on a given dataset. Empirically, we find that even under a simple defense, the MNIST-1-7 and Dogfish datasets are resilient to attack, while in contrast the IMDB sentiment dataset can be driven from 12% to 23% test error by adding only 3% poisoned data.
The paper studies an interesting topic and is overall nice to read. However, I have doubts about the correctness of the main theorem (Proposition 1), which makes two statements: one that bounds the max-min loss M from below and above, and a second that further bounds the upper bound. While the second claim is proven in the appendix; the first is not. The upper bound follows from the convexity of U and Equation 5. (It might be helpful for the reader to explicitly state this finding and argue why D_p reduces to a set of \epsilon n identical points.) However, I can't see why the lower bound holds. Computing M requires the solution of a saddle point problem. If one knew one part of this saddle point solution, one could easily bound M from above and/or below. However, Algorithm 1 only generates interim solutions that eventually converge to the saddle point. Concretely, in (6), D_p is not (necessarily) part of the saddle point solution; and \hat theta is neither the minimizer of L w.r.t. D_c \cup D_p nor part of the saddle point solution. So why is this necessarily a lower bound?
It's hard to judge the quality of candidate set D_p. One could argue that it would be better to actually run the algorithm till convergence and take the final data pair (x, y) \epsilon n times. Alternatively, one could take the last \epsilon n data points before the algorithm converges. Why the first \epsilon n update steps give a better candidate set is unclear to me. Whether the resulting model after \epsilon n update steps is any close to an optimum is unclear to me, too. Overall, there is little theoretical justification for the generation of the candidate attack in Alg. 1; it is for sure not the solution of (4). Hence, it's hard to conclude something like "even under a simple defense, the XYZ datasets are resilient to attack". And since the attack is model-based - it is constructed with a certain model (in this case, a linear model), loss function, and regularization scheme in mind - it is also hard to conclude "the ABC dataset can be driven from 12% to 23% test error"; this might be true for the concretely chosen model, but a different model might be more robust to this kind of poising attack.
To me the value of the approach is that it allows to study the sensitivity of a concrete classifier (e.g., SVM) w.r.t. a rather broad set of attacks. In case of such sensitivity (for a concrete dataset), one could think about choosing a different defense model and/or classifier. However, in the absence of such sensitivity one should NOT conclude the robustness of the model.
In Section 3.2, the defenses are defined in terms of class means rather than centroids (as in (2)).
Eq. (8) requires F(\pi), though, F is defined over a set of data points. I don't fully understand this maximization term and I would like the authors to provide more details how to generalize Alg. 1 to this setting.
In the experimentation section, some details are missing to reproduce the results; for example, the algorithm has parameters \roh and \eta, the defense strategies have thresholds. It is neither clear to me how those parameters are chosen, nor which parameters were used in the experiments.
After authors response:
Regarding the correctness of correctness of Prop. 1: \hat {theta} is the minimizer of L w.r.t. D_c \cup D_p; but D_c \cup D_p is not simultaneously a maximizer of L. M states the value of L at a saddle point, but neither \hat{theta} nor D_c \cup D_p is necessarily part of a saddle point. However, it became clear to me that this is not required. We can replace
max_D min_\theta L(D, \theta)
by
max_D L(D, \hat
{\theta}_D)
where \hat{theta}_D is the minimizer w.r.t. D. Of course, this quantity is always smaller than M := L(\hat{D}, \hat{\theta}_\hat{D}) for any maximizer \hat{D} and saddle point (\hat{D}, \hat{\theta}_\hat{D}). I got confused from the two meanings of D_p, which is used as a free variable in the maximization in (4) and as the result of Alg. 1.