Feature Cross-Substitution in Adversarial Classification
The success of machine learning, particularly in supervised settings, has led to numerous attempts to apply it in adversarial settings such as spam and malware detection. The core challenge in this class of applications is that adversaries are not static data generators, but make a deliberate effort to evade the classifiers deployed to detect them. We investigate both the problem of modeling the objectives of such adversaries, as well as the algorithmic problem of accounting for rational, objective-driven adversaries. In particular, we demonstrate severe shortcomings of feature reduction in adversarial settings using several natural adversarial objective functions, an observation that is particularly pronounced when the adversary is able to substitute across similar features (for example, replace words with synonyms or replace letters in words). We offer a simple heuristic method for making learning more robust to feature cross-substitution attacks. We then present a more general approach based on mixed-integer linear programming with constraint generation, which implicitly trades off overfitting and feature selection in an adversarial setting using a sparse regularizer along with an evasion model. Our approach is the first method for combining an adversarial classification algorithm with a very general class of models of adversarial classifier evasion. We show that our algorithmic approach significantly outperforms state-of-the-art alternatives.
This paper tackles the problem of classification in an adversarial setting, specifically in the case of spam and phishing detection.  It presents a class of adversaries that use "feature cross-substitution" to evade classification while still accomplishing their attacks as well as two approaches to handling this type of adversary: a heuristic one based on feature equivalence classes and a more principled one based on a Stackleberg game solved with integer/linear programming.Positive Points:1.  The paper is well-written and well-motivated, with a good coverage of the related literature.  One possible bit of related work that might be included is that of Karlberger (et al.?) on Penetrating Bayesian Spam Filters.2. The idea of a complementary form of classification error (c.f. bias and variance) in adversarial learning settings is nice.  Probably not an earth shatteringly new/surprising idea, but it is nice to see people thinking about this kind of thing.3. Empirical results on three different email data sets, compared to state-of-the-art methods, are impressive.Points of Concern:4. In Sec. 4, is the simple model of 1- and 2-letter substitutions a reasonable approximation for more sophisticated types of substitution attacks?5. In Sec. 4, the first key observation, that "when the adversary uses either substitution cost function, the optimal number of features increases", while intuitively appealing, doesn't seem to me to be demonstrated very clearly at all in the reported experiments.  The claimed "right shift" does not seem obvious to me in the majority of the results shown in Fig. 2.  Indeed, if I were looking at these data in an unbiased way, I'd be tempted to say they show that in fact there is no shift in the majority of cases and, therefore, there is no noticeable tradeoff between overfitting and feature selection under adversarial conditions.  Which in turns lead to the question of whether this proposed third type of error really exists.  The rest of the paper is pretty clear and convincing (and the empirical results certainly seem to show that something is going on) -- am I missing something here?6. In Sec. 4, the second key observation seems obvious on the one hand---of course evasion will account for a large amount of error if there is a lot of malicious activity---and, at the same time, I wonder if we can accurately make this claim; that is, are we confident that the bias and variance components of the error really are small under this scenario? 7. Seems like the last paragraph of Sec. 6 is hiding lots and lots and lots of details.  I realize NIPS has a tight page limit, but it seems like a bit more here, traded with a bit less somewhere else would make this a better paper.Small points:8. References could be tidied up, particularly capitalization issues.9. In Sec. 2, subheading "The Game", should "the defender" in bullet 3 be reconciled with the "learner" in bullet 1?10. I'm not clear on what exactly the notation $x^A_j \oplus x^'_j$ means.11. In Fig. 1, right hand side, does the x-axis label "# Features" refer to the size of the equivalence class (it seems to be hinted that it does in the text, but this should be clarified).12. There are a few places throughout the paper where there are usage issues with articles (mostly they are occasionally missing).13. In Sec. 4, when details on the data sets are given, class balance is described for both the Enron and UCI data sets but is neglected for the Ling data set.14. In Sec. 6, I had a hard time with the definition, $e_j = \Sum…$, where $e_j = w^T…$.  Is this just an odd way to say how we are going to compute the $w^TA()$ term in Eq. 4? Or, is something else being said here (with a possible typo obfuscating it)?   Can this be clarified somehow?Interesting questions:15. With a large enough cost budget, it would seem a foregone conclusion that the defender will have little hope of accurately identifying spam while also allowing legitimate email through.  However, it would also seem that at some threshold of substitution, spam effectiveness will begin to really suffer (that is, not even really gullible people will give it any credence at some point).  This could be modeled with a non-linear cost function or with a simple threshold, for example, but do we have any way to bound this effective cost budget?  That is, is there a point at which we don't have to worry about detecting it because no adversary would choose such a highly substituted attack vector?