Blind Attacks on Machine Learners
The importance of studying the robustness of learners to malicious data is well established. While much work has been done establishing both robust estimators and effective data injection attacks when the attacker is omniscient, the ability of an attacker to provably harm learning while having access to little information is largely unstudied. We study the potential of a "blind attacker" to provably limit a learner's performance by data injection attack without observing the learner's training set or any parameter of the distribution from which it is drawn. We provide examples of simple yet effective attacks in two settings: firstly, where an "informed learner" knows the strategy chosen by the attacker, and secondly, where a "blind learner" knows only the proportion of malicious data and some family to which the malicious distribution chosen by the attacker belongs. For each attack, we analyze minimax rates of convergence and establish lower bounds on the learner's minimax risk, exhibiting limits on a learner's ability to learn under data injection attack even when the attacker is "blind".
This paper abstracts a problem of interest in (applied) adversarial machine learning, whereby an adversary manipulates a learner trained partially on benign stochastic data, with adversarially-chosen, stochastic data. Specifically the paper studies a learner intending to estimate parameter (theta) for distribution (F) from which training data is drawn, and a blind attacker – not knowing the distribution or the training set – who injects data with from a different distribution (G(phi)), in order to maximise the minimax risk for the learner. The authors discuss two settings – “informed learner” and “blind learner”, and investigate the effect of the relationship (KL divergence) between F(theta) and G(phi).
The paper presents an appropriate theoretical treatment of an interesting question that has persisted in adversarial machine learning: that of the effect of a blind attacker poisoning training data without access to benign iid training data. The paper's approach is one of minimax theory, with analysis variations given for a defender/learner aware of the attacker's poison distribution, and (the more realistic) case of a blind defender who is unaware of the attacker's choice. Results depend on the KL divergence between benign and malicious data distributions.  There have been relatively few theoretical treatments of adversarial machine learning settings. The authors suitably note the recent work of Duchi et al. on minimax rates and differential privacy (that appeared recently at NIPS), with this paper being complementary.  However some issues remain:  The work on Kearns and Li (1993) who looked at the effect of a fraction of contaminated data in the PAC learning framework should be cited and related briefly. Similarly the vast body of work on robust statistics: the presented framework seems to not be unlike the gross-error model (sampling distribution is a mixture of benign and a contamination distribution; the minimax approach there is to consider a contamination distribution that is worst case; this yields Huber estimators that is a robust class of estimators in the setting; etc.)  The paper could be better structured. For example, Section 1 spans half of the paper and would be better organised if divided into 2- 3 sections.  While the paper is written appropriately for a specialised COLT sub-audience present at NIPS, since NIPS is broader (than ever), and the (more applied) adversarial learning community should find the paper interesting, it would be beneficial to rework the exposition and background with such readers in mind. For example detailing the connection between problems at #89  Several notations are used before being properly defined, e.g., delta in #199, Vol(Z) in #292 (instead, these are defined in the supplementary material).  Not sure Fig1,2 strictly represent Markov chains (typically identical transition probabilities across finite states, etc.) but no significant matter. Also, consider shading nodes that are observed.  Quotations leading quoted phrases should be opening quotation marks. Something with the typesetting (#161) seems broken with the definition of theta in relation to theta_i.  **After author response: Thankyou for the thoughtful response. The assurances that connections with robust stats and PAC will be added as appropriate, are well received. The initial explanation on the differences between the blind model and the typical (non-agnostic) model typical in robustness appear to make sense.