HERE’S MY POINT: ARGUMENTATION MINING WITH POINTER NETWORKS
One of the major goals in automated argumentation mining is to uncover the argument structure present in argumentative text. In order to determine this structure, one must understand how different individual components of the overall argument are linked. General consensus in this field dictates that the argument components form a hierarchy of persuasion, which manifests itself in a tree structure. This work provides the first neural network-based approach to argumentation mining, focusing on extracting links between argument components, with a secondary focus on classifying types of argument components. In order to solve this problem, we propose to use a modification of a Pointer Network architecture. A Pointer Network is appealing for this task for the following reasons: 1) It takes into account the sequential nature of argument components; 2) By construction, it enforces certain properties of the tree structure present in argument relations; 3) The hidden representations can be applied to auxiliary tasks. In order to extend the contribution of the original Pointer Network model, we construct a joint model that simultaneously attempts to learn the type of argument component, as well as continuing to predict links between argument components. The proposed model achieves state-of-the-art results on two separate evaluation corpora. Furthermore, our results show that optimizing for both tasks, as well as adding a fully-connected layer prior to recurrent neural network input, is crucial for high performance.
1 INTRODUCTION
Computational approaches to argument mining/understanding have become very popular (Persing & Ng, 2016; Cano-Basave & He, 2016; Wei et al., 2016; Ghosh et al., 2016; Palau & Moens, 2009; Habernal & Gurevych, 2016). One important avenue in this work is to understand the structure in argumentative text (Persing & Ng, 2016; Peldszus & Stede, 2015; Stab & Gurevych, 2016; Nguyen & Litman, 2016). One fundamental assumption when working with argumentative text is the presence of Arguments Components (ACs). The types of ACs are generally characterized as a claim or a premise (Govier, 2013), with premises acting as support (or possibly attack) units for claims. To model more complex structures of arguments, some annotation schemes also include a major claim AC type (Stab & Gurevych, 2016; 2014b).
Generally, the task of processing argument structure encapsulates four distinct subtasks: 1) Given a sequence of tokens that represents an entire argumentative text, determine the token subsequences that constitute non-intersecting ACs; 2) Given an AC, determine the type of AC (claim, premise, etc.); 3) Given a set/list of ACs, determine which ACs have a link that determine overall argument structure; 4) Given two linked ACs, determine whether the link is of a supporting or attacking relation. In this work, we focus on subtasks 2 and 3.
There are two key assumptions our work makes going forward. First, we assume subtask 1 has been completed, i.e. ACs have already been identified. Second, we follow previous work that assumes a tree structure for the linking of ACs (Palau & Moens, 2009; Cohen, 1987; Peldszus & Stede, 2015; Stab & Gurevych, 2016) Specifically, a given AC can only have a single outgoing link, but can have numerous incoming links. Furthermore, there is a ‘head’ component that has
First, [ :::::: cloning :::: will ::: be ::::::::: beneficial :::: for :::: many :::::: people ::::: who ::: are :: in :::: need ::: of ::::: organ ::::::::: transplants]AC1. In addition, [it shortens the healing process]AC2. Usually, [it is very rare to find an appropriate organ donor]AC3 and [by using cloning in order to raise required organs the waiting time can be shortened tremendously]AC4.
AC1 Claim
AC2 Premise
AC4 Premise AC3 Premise
Figure 1: An example of argument structure with four ACs. The left side shows raw text that has been annotated for the presence of ACs. Squiggly and straight underlining means an AC is a claim or premise, respectively. The ACs in the text have also been annotated for links to other ACs, which is show in the right figure. ACs 3 and 4 are premises that link to another premise, AC2. Finally, AC2 links to a claim, AC1. AC1 therefore acts as the central argumentative component.
no outgoing link (the top of the tree). Figure 1 shows an example that we will use throughout the paper to concretely explain how our approach works. First, the left side of the figure presents the raw text of a paragraph in a persuasive essay (Stab & Gurevych, 2016), with the ACs contained in square brackets. Squiggly verse straight underlining differentiates between claims and premises, respectively. The ACs have been annotated as to how the ACs are linked, and the right side of the figure reflects this structure. The argument structure with four ACs forms a tree, where AC2 has two incoming links, and AC1 acts as the head, with no outgoing links. We also specify the type of AC, with the head AC marked as claim and the remaining ACs marked as premise. Lastly, we note that the order of arguments components can be a strong indicator of how components should related. Linking to the first argument component can provide a competitive baseline heuristic (Peldszus & Stede, 2015; Stab & Gurevych, 2016).
Given the task at hand, we propose a modification of a Pointer Network (PN) (Vinyals et al., 2015b). A PN is a sequence-to-sequence model that outputs a distribution over the encoding indices at each decoding timestep. The PN is a promising model for link extraction in argumentative text because it inherently possesses three important characteristics: 1) it is able to model the sequential nature of ACs; 2) it constrains ACs to have a single outgoing link, thus partly enforcing the tree structure; 3) the hidden representations learned by the model can be used for jointly predicting multiple subtasks. We also note that since a PN is a type of sequence-to-sequence model (Sutskever et al., 2014), it allows the entire sequence to be seen before making prediction. This is important because if the problem were to be approached as standard sequence modeling (Graves & Schmidhuber, 2009; Robinson, 1994), making predictions at each forward timestep, it would only allow links to ACs that have already been seen. This is equivalent to only allowing backward links. We note that we do test a simplified model that only uses hidden states from an encoding network to make predictions, as opposed to the sequence-to-sequence architecture present in the PN (see Section 5).
PNs were originally proposed to allow a variable length decoding sequence (Vinyals et al., 2015b). Alternatively, the PN we implement differs from the original model in that we decode for the same number of timesteps as there are input components. We also propose a joint PN for both extracting links between ACs and predicting the type of AC. The model uses the hidden representation of ACs produced during the encoding step (see Section 3.4). Aside from the partial assumption of tree structure in the argumentative text, our models do not make any additional assumptions about the AC types or connectivity, unlike the work of Peldszus (2014). We evaluate our models on the corpora of Stab & Gurevych (2016) and Peldszus (2014), and compare our results with the results of the aformentioned authors.
2 RELATED WORK
Recent work in argumentation mining offers data-driven approaches for the task of predicting links between ACs. Stab & Gurevych (2014b) approach the task as a binary classification problem. The
authors train an SVM with various semantic and structural features. Peldszus & Stede (2015) have also used classification models for predicting the presence of links. Various authors have also proposed to jointly model link extraction with other subtasks from the argumentation mining pipeline, using either an Integer Linear Programming (ILP) framework (Persing & Ng, 2016; Stab & Gurevych, 2016) or directly feeding previous subtask predictions into another model. The former joint approaches are evaluated on annotated corpora of persuasive essays (Stab & Gurevych, 2014a; 2016), and the latter on a corpus of microtexts (Peldszus, 2014). The ILP framework is effective in enforcing a tree structure between ACs when predictions are made from otherwise naive base classifiers.
Unrelated to argumentation mining specifically, recurrent neural networks have previously been proposed to model tree/graph structures in a linear manner. Vinyals et al. (2015c) use a sequenceto-sequence model for the task of syntactic parsing. The authors linearize input parse graphs using a depth-first search, allowing it to be consumed as a sequence, achieving state-of-the-art results on several syntactic parsing datasets. Bowman et al. (2015) experiment on an artificial entailment dataset that is specifically engineered to capture recursive logic (Bowman et al., 2014). The text is annotated with brackets, in an original attempt to provide easy input into a recursive neural network. However, standard recurrent neural networks can take in complete sentence sequences, brackets included, and perform competitively with a recursive neural network.
3 POINTER NETWORK FOR LINK EXTRACTION
In this section we will describe how we use a PN for the problem of extracting links between ACs. We begin by giving a general description of the PN model.
3.1 POINTER NETWORK
A PN is a sequence-to-sequence model (Sutskever et al., 2014) with attention (Bahdanau et al., 2014) that was proposed to handle decoding sequences over the encoding inputs, and can be extended to arbitrary sets (Vinyals et al., 2015a). The original motivation for a pointer network was to allow networks to learn solutions to algorithmic problems, such as the traveling salesperson and convex hull, where the solution is a sequence over candidate points. The PN model is trained on input/output sequence pairs (E,D), where E is the source and D is the target (our choice of E,D is meant to represent the encoding, decoding steps of the sequence-to-sequence model). Given model parameters Θ, we apply the chain rule to determine the probability of a single training example:
p(D|E; Θ) = m(E)∏ i=1 p(Di|D1, ..., Di−1, E; Θ) (1)
where the functionm signifies that the number of decoding timesteps is a function of each individual training example. We will discuss shortly why we need to modify the original definition of m for our application. By taking the log-likelihood of Equation 1, we arrive at the optimization objective:
Θ∗ = arg max Θ ∑ E,D log p(D|E; Θ) (2)
which is the sum over all training example pairs.
The PN uses Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) for sequential modeling, which produces a hidden layer h at each encoding/decoding timestep. In practice, the PN has two separate LSTMs, one for encoding and one for decoding. Thus, we refer to encoding hidden layers as e, and decoding hidden layers as d.
The PN uses a form of content-based attention (Bahdanau et al., 2014) to allow the model to produce a distribution over input elements. This can also be thought of as a distribution over input indices, wherein a decoding step ‘points’ to the input. Formally, given encoding hidden states (e1, ..., en), The model calculates p(Di|D1, ..., Di−1, E) as follows:
uij = v T tanh(W1ej +W2di) (3)
p(Di|D1, ..., Dj−1, E) = softmax(ui) (4) where matrices W1, W2 and vector v are parameters of the model (along with the LSTM parameters used for encoding and decoding). In Equation 3, prior to taking the dot product with v, the resulting transformation can be thought of as creating a joint, hidden representation of inputs i and j. Vector ui in equation 4 is of length n, and index j corresponds to input element j. Therefore, by taking the softmax of ui, we are able to create a distribution over the input.
3.2 LINK EXTRACTION AS SEQUENCE MODELING
A given piece of text has a set of ACs, which occur in a specific order in the text, (C1, ..., Cn). Therefore, at encoding timestep i, the model is fed a representation of Ci. Since the representation is large and sparse (see Section 3.3 for details on how we represent ACs), we add a fully-connected layer before the LSTM input. Given a representation Ri for AC Ci the LSTM input Ai becomes:
Ai = σ(WrepRi + brep) (5)
where Wrep, brep in turn become model parameters, and σ is the sigmoid function1. (similarly, the decoding network applies a fully-connected layer with sigmoid activation to its inputs, see Figure 3). At encoding step i, the encoding LSTM produces hidden layer ei, which can be thought of as a hidden representation of AC Ci.
In order to make the PN applicable to the problem of link extraction, we explicitly set the number of decoding timesteps to be equal to the number of input components. Using notation from Equation 1, the decoding sequence length for an encoding sequence E is simply m(E) = |{C1, ..., Cn}|, which is trivially equal to n. By constructing the decoding sequence in this manner, we can associate decoding timestep i with AC Ci.
From Equation 4, decoding timestep Di will output a distribution over input indices. The result of this distribution will indicate to which AC component Ci links. Recall there is a possibility that an AC has no outgoing link, such as if it’s the root of the tree. In this case, we state that if AC Ci does not have an outgoing link, decoding step Di will output index i. Conversely, if Di outputs index j, such that j is not equal to i, this implies that Ci has an outgoing link to Cj . For the argument structure in Figure 1, the corresponding decoding sequence is (1, 1, 2, 2). The topology of this decoding sequence is illustrated in Figure 2. Note how C1 points to itself since it has no outgoing link.
Finally, we note that we modify the PN structure to have a Bidirectional LSTM as the encoder. Thus, ei is the concatenation of forward and backward hidden states −→e i and ←−e n−i+1, produced by two separate LSTMs. The decoder remains a standard forward LSTM.
3.3 REPRESENTING ARGUMENT COMPONENTS
At each timestep of the decoder, the network takes in the representation of an AC. Each AC is itself a sequence of tokens, similar to the recently proposed Question-Answering dataset (Weston et al., 2015). We follow the work of Stab & Gurevych (2016) and focus on three different types of features
1We also experimented with relu and elu activations, but found sigmoid to yeild the best performance.
to represent our ACs: 1) Bag-of-Words of the AC; 2) Embedding representation based on GloVe embeddings (Pennington et al., 2014); 3) Structural features: Whether or not the AC is the first AC in a paragraph, and Whether the AC is in an opening, body, or closing paragraph. See Section 6 for an ablation study of the proposed features.
3.4 JOINT NEURAL MODEL
Up to this point, we focused on the task of extracting links between ACs. However, recent work has shown that joint models that simultaneously try to complete multiple aspects of the subtask pipeline outperform models that focus on a single subtask (Persing & Ng, 2016; Stab & Gurevych, 2014b; Peldszus & Stede, 2015). Therefore, we will modify the architecture we proposed in Section 3 so that it would allow us to perform AC classification (Kwon et al., 2007; Rooney et al., 2012) together with link prediction. Knowledge of an individual subtask’s predictions can aid in other subtasks. For example, claims do not have an outgoing link, so knowing the type of AC can aid in the link prediction task. This can be seen as a way of regularizing the hidden representations from the encoding component (Che et al., 2015).
Predicting AC type is a straightforward classification task: given AC Ci, we need to predict whether it is a claim or premise. Some annotation schemes also include the class major claim (Stab & Gurevych, 2014a), which means this can be a multi-class classification task. For encoding timestep i, the model creates hidden representation ei. This can be thought of as a representation of AC Ci. Therefore, our joint model will simply pass this representation through a fully connected layer as follows: zi = Wclsei + bcls (6) where Wcls, bcls become elements of the model parameters, Θ. The dimensionality of Wcls, bcls is determined by the number of classes. Lastly, we use softmax to form a distribution over the possible classes.
Consequently, the probability of predicting component type at timestep i is defined as:
p(Ci) = p(Ei| −→ E i, ←− E i; Θ) (7)
p(Ei| −→ E i, ←− E i; Θ) = softmax(zi) (8)
Finally, combining this new prediction task with Equation 2, we arrive at the new training objective:
Θ∗ = arg max Θ α ∑ E,D log p(D|E; Θ) + (1− α) ∑ E log p(E|Θ) (9)
which simply sums the costs of the individual prediction tasks, and the second summation is the cost for the new task of predicting argument component type. α ∈ [0, 1] is a hyperparameter that
specifies how we weight the two prediction tasks in our cost function. The architecture of the joint model, applied to our ongoing example, is illustrated in Figure 3.
4 EXPERIMENTAL DESIGN
As we have previously mentioned, our work assumes that ACs have already been identified. That is, the token sequence that comprises a given AC is already known. The order of ACs corresponds directly to the order in which the ACs appear in the text. Since ACs are non-overlapping, there is no ambiguity in this ordering. We test the effectiveness of our proposed model on a dataset of persuasive essays (Stab & Gurevych, 2016), as well as a dataset of microtexts (Peldszus, 2014). The feature space for the persuasive essay corpus has roughly 3,000 dimensions, and the microtext corpus feature space has between 2,500 and 3,000 dimensions, depending on the data split (see below).
The persuasive essay corpus contains a total of 402 essays, with a frozen set of 80 essays held out for testing. There are three AC types in this corpus: major claim, claim, and premise. We follow the creators of the corpus and only evaluate ACs within a given paragraph. That is, each training/test example is a sequence of ACs from a paragraph. This results in a 1,405/144 training/test split. The microtext corpus contains 112 short texts. Unlike, the persuasive essay corpus, each text in this corpus is itself a complete example. Since the dataset is small, the authors have created 10 sets of 5-fold cross-validation, reporting the the average across all splits for final model evaluation. This corpus contains only two types of ACs (claim and premise) The annotation of argument structure of the microtext corpus varies from the persuasive essay corpus; ACs can be linked to other links, as opposed to ACs. Therefore, if AC Ci is annotated to be linked to link l, we create a link to the source AC of l. On average, this corpus has 5.14 ACs per text. Lastly, we note that predicting the presence of links is directional (ordered): predicting a link between the pair Ci, Cj(i 6= j) is different than Cj , Ci.
We implement our models in TensorFlow (Abadi et al., 2015). Our model has the following parameters: hidden input dimension size 512, hidden layer size 256 for the bidirectional LSTMs, hidden layer size 512 for the LSTM decoder, α equal to 0.5, and dropout (Srivastava et al., 2014) of 0.9. We believe the need for such high dropout is due to the small amounts of training data (Zarrella & Marsh, 2016), particularly in the Microtext corpus. All models are trained with Adam optimizer (Kingma & Ba, 2014) with a batch size of 16. For a given training set, we randomly select 10% to become the validation set. Training occurs for 4,000 epochs. Once training is completed, we select the model with the highest validation accuracy (on the link prediction task) and evaluate it on the held-out test set. At test time, we take a greedy approach and select the index of the probability distribution (whether link or type prediction) with the highest value.
5 RESULTS
The results of our experiments are presented in Tables 1 and 2. For each corpus, we present f1 scores for the AC type classification experiment, with a macro-averaged score of the individual class f1 scores. We also present the f1 scores for predicting the presence/absence of links between ACs, as well as the associated macro-average between these two values.
We implement and compare four types of neural models: 1) The previously described PN-based model depicted in Figure 3 (called PN in the tables); 2) The same as 1), but without the fullyconnected input layers; 3) The same as 1), but the model only predicts the link task, and is therefore not optimized for type prediction; 4) A non-sequence-to-sequence model that uses the hidden layers produced by the BLSTM encoder with the same type of attention as the PN (called BLSTM in the table). That is, di in Equation 3 is replaced by ei.
In both corpora we compare against the following previously proposed models: Base Classifier (Stab & Gurevych, 2016) is feature-rich, task-specific (AC type or link extraction) SVM classifier. Neither of these classifiers enforce structural or global constraints. Conversely, the ILP Joint Model (Stab & Gurevych, 2016) provides constrains by sharing prediction information between the base classifier. For example, the model attempts to enforce a tree structure among ACs within a given paragraph, as well as using incoming link predictions to better predict the type class claim. For the
microtext corpus only, we have the following comparative models: Simple (Peldszus & Stede, 2015) is a feature-rich logistic regression classifier. Best EG (Peldszus & Stede, 2015) creates an Evidence Graph (EG) from the predictions of a set of base classifier. The EG models the potential argument structure, and offers a global optimization objective that the base classifiers attempt to optimize by adjusting their individual weights. Lastly, MP+p (Peldszus & Stede, 2015) combines predictions from base classifiers with a MSTParser, which applies 1-best MIRA structured learning.
6 DISCUSSION
First, we point out that the PN model achieves state-of-the-art on 10 of the 13 metrics in Tables 1 and 2, including the highest results in all metrics on the Persuasive Essay corpus, as well as link prediction on the Microtext corpus. The performance on the Microtext corpus is very encouraging for several reasons. First, the fact that the model can perform so well with only a hundred training examples is rather remarkable. Second, although we motivate the use of a PN due to the fact that it partially enforces the tree structure in argumentation, other models explicitly contain further constraints. For example, only premises can have outgoing links, and there can be only one claim in an AC. As for the other neural models, the BLSTM model performs competitively with the ILP Joint Model on the persuasive essay corpus, but trails the performance of the PN model. We believe this is because the PN model is able to create two different representations for each AC, one each in the encoding/decoding state, which benefits performance in the dual tasks, whereas the BLSTM model must encode information relating to type as well as link prediction in a single hidden representation. On one hand, the BLSTM model outperforms the ILP model on link prediction, yet it is not able to match the ILP Joint Model’s performance on type prediction, primarily due to the BLSTM’s poor performance on predicting the major claim class. Another interesting outcome is the importance of the fully-connected layer before the LSTM input. The results show that this extra layer of depth is crucial for good performance on this task. Without it, the PN model is only able to perform competitively with the Base Classifier. The results dictate that even a simple fully-connected layer with sigmoid activation can provide a useful dimensionality reduction for feature representation. Finally, the PN model that only extracts links suffers a large drop in performance, conveying that the joint aspect of the PN model is crucial for high performance in the link prediction task.
Table 3 shows the results of an ablation study for AC feature representation. Regarding link prediction, BOW features are clearly the most important, as their absence results in the highest drop in performance. Conversely, the presence of structural features provides the smallest boost in performance, as the model is still able to record state-of-the-art results compared to the ILP Joint Model. This shows that, one one hand, the PN model is able to capture structural ques through sequence
modeling and semantics (the ILP Joint Model directly integrates these structural features), however the PN model still does benefit from their explicit presence in the feature representation. When considering type prediction, both BOW and structural features are important, and it is the embedding features that provide the least benefit. The Ablation results also provide an interesting insight into the effectiveness of different ‘pooling’ strategies for using individual token embeddings to create a multi-word embedding. The popular method of averaging embeddings (which is used by Stab & Gurevych (2016) in their system) is in fact the worst method, although its performance is still competitive with the previous state-of-the-art. Conversely, max pooling produces results that are on par with the PN results from Table 1.
Table 4 shows the results on the Persuasive Essay test set with the examples binned by sequence length. First, it is not a surprise to see that the model performs best when the sequences are the shortest. As the sequence length increases, the accuracy on link prediction drops. This is possibly due to the fact that as the length increases, a given AC has more possibilities as to which other AC it can link to, making the task more difficult. Conversely, there is actually a rise in no link prediction accuracy from the second to third row. This is likely due to the fact that since the model predicts at most one outgoing link, it indirectly predicts no link for the remaining ACs in the sequence. Since the chance probability is low for having a link between a given AC in a long sequence, the no link performance is actually better in longer sequences.
7 CONCLUSION
In this paper we have proposed how to use a modified PN (Vinyals et al., 2015b) to extract links between ACs in argumentative text. We evaluate our models on two corpora: a corpus of persuasive essays (Stab & Gurevych, 2016), and a corpus of microtexts (Peldszus, 2014). The PN model records state-of-the-art results on the persuasive essay corpus, as well as achieving state-of-the-art results for link prediction on the microtext corpus, despite only having 90 training examples. The results show that jointly modeling the two prediction tasks is crucial for high performance, as well as the presence of a fully-connected layer prior to the LSTM input. Future work can attempt to learn the AC representations themselves, such as in Kumar et al. (2015). Lastly, future work can integrate subtasks 1 and 4 into the model. The representations produced by Equation 3 could potentially be used to predict the type of link connecting ACs, i.e. supporting or attacking; this is the fourth subtask in the pipeline. In addition, a segmenting technique, such as the one proposed by Weston et al. (2014), can accomplish subtask 1.

This paper addresses the problem of argument mining, which consists of finding argument types and predicting the relationships between the arguments. The authors proposed a pointer network structure to recover the argument relations. They also propose modifications on pointer network to perform joint training on both type and link prediction tasks. Overall the model is reasonable, but I am not sure if ICLR is the best venue for this work.
My first concern of the paper is on the novelty of the model. Pointer network has been proposed before. The proposed multi-task learning method is interesting, but the authors only verified it on one task. This makes me feel that maybe the submission is more for a NLP conference rather than ICLR.
The authors stated that the pointer network is less restrictive compared to some of the existing tree predicting method. However, the datasets seem to only contain single trees or forests, and the stack-based method can be used for forest prediction by adding a virtual root node to each example (as done in the dependency parsing tasks). Therefore, I think the experiments right now cannot reflect the advantages of pointer network models unfortunately.
My second concern of the paper is on the target task. Given that the authors want to analyze the structures between sentences, is the argumentation mining the best dataset? For example, authors could verify their model by applying it to the other tasks that require tree structures such as dependency parsing. As for NLP applications, I found that the assumption that the boundaries of AC are given is a very strong constraint, and could potentially limit the usefulness of the proposed model.
Overall, in terms of ML, I also feel that baseline methods the authors compared to are probably strong for the argument mining task, but not necessary strong enough for the general tree/forest prediction tasks (as there are other tree/forest prediction methods). In terms of NLP applications, I think the assumption of having AC boundaries is too restrictive, and maybe ICLR is not the best venture for this submission.


ON DETECTING ADVERSARIAL PERTURBATIONS
Machine learning and deep learning in particular has advanced tremendously on perceptual tasks in recent years. However, it remains vulnerable against adversarial perturbations of the input that have been crafted specifically to fool the system while being quasi-imperceptible to a human. In this work, we propose to augment deep neural networks with a small “detector” subnetwork which is trained on the binary classification task of distinguishing genuine data from data containing adversarial perturbations. Our method is orthogonal to prior work on addressing adversarial perturbations, which has mostly focused on making the classification network itself more robust. We show empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans. Moreover, while the detectors have been trained to detect only a specific adversary, they generalize to similar and weaker adversaries. In addition, we propose an adversarial attack that fools both the classifier and the detector and a novel training procedure for the detector that counteracts this attack.
1 INTRODUCTION
In the last years, machine learning and in particular deep learning methods have led to impressive performance on various challenging perceptual tasks, such as image classification (Russakovsky et al., 2015; He et al., 2016) and speech recognition (Amodei et al., 2016). Despite these advances, perceptual systems of humans and machines still differ significantly. As Szegedy et al. (2014) have shown, small but carefully directed perturbations of images can lead to incorrect classification with high confidence on artificial systems. Yet, for humans these perturbations are often visually imperceptible and do not stir any doubt about the correct classification. In fact, so called adversarial examples are crucially characterized by requiring minimal perturbations that are quasi-imperceptible to a human observer. For computer vision tasks, multiple techniques to create such adversarial examples have been developed recently. Perhaps most strikingly, adversarial examples have been shown to transfer between different network architectures, and networks trained on disjoint subsets of data (Szegedy et al., 2014). Adversarial examples have also been shown to translate to the real world (Kurakin et al., 2016), e.g., adversarial images can remain adversarial even after being printed and recaptured with a cell phone camera. Moreover, Papernot et al. (2016a) have shown that a potential attacker can construct adversarial examples for a network of unknown architecture by training an auxiliary network on similar data and exploiting the transferability of adversarial inputs.
The vulnerability to adversarial inputs can be problematic and even prevent the application of deep learning methods in safety- and security-critical applications. The problem is particularly severe when human safety is involved, for example in the case of perceptual tasks for autonomous driving. Methods to increase robustness against adversarial attacks have been proposed and range from augmenting the training data (Goodfellow et al., 2015) over applying JPEG compression to the input (Dziugaite et al., 2016) to distilling a hardened network from the original classifier network (Papernot et al., 2016b). However, for some recently published attacks (Carlini & Wagner, 2016), no effective counter-measures are known yet.
In this paper, we propose to train a binary detector network, which obtains inputs from intermediate feature representations of a classifier, to discriminate between samples from the original data set and adversarial examples. Being able to detect adversarial perturbations might help in safety- and security-critical semi-autonomous systems as it would allow disabling autonomous operation and
requesting human intervention (along with a warning that someone might be manipulating the system). However, it might intuitively seem very difficult to train such a detector since adversarial inputs are generated by tiny, sometimes visually imperceptible, perturbations of genuine examples. Despite this intuition, our results on CIFAR10 and a 10-class subset of ImageNet show that a detector network that achieves high accuracy in detection of adversarial inputs can be trained successfully. Moreover, while we train a detector network to detect perturbations of a specific adversary, our experiments show that detectors generalize to similar and weaker adversaries. An obvious attack against our approach would be to develop adversaries that take into account both networks, the classification and the adversarial detection network. We present one such adversary and show that we can harden the detector against such an adversary using a novel training procedure.
2 BACKGROUND
Since their discovery by Szegedy et al. (2014), several methods to generate adversarial examples have been proposed. Most of these methods generate adversarial examples by optimizing an image w.r.t. the linearized classification cost function of the classification network by maximizing the probability for all but the true class or minimizing the probability of the true class (e.g., (Goodfellow et al., 2015), (Kurakin et al., 2016)). The method introduced by Moosavi-Dezfooli et al. (2016b) estimates a linearization of decision boundaries between classes in image space and iteratively shifts an image towards the closest of these linearized boundaries. For more details about these methods, please refer to Section 3.1.
Several approaches exist to increase a model’s robustness against adversarial attacks. Goodfellow et al. (2015) propose to augment the training set with adversarial examples. At training time, they minimize the loss for real and adversarial examples, while adversarial examples are chosen to fool the current version of the model. In contrast, Zheng et al. (2016) propose to append a stability term to the objective function, which forces the model to have similar outputs for samples of the training set and their perturbed versions. This differs from data augmentation since it encourages smoothness of the model output between original and distorted samples instead of minimizing the original objective on the adversarial examples directly. Another defense-measure against certain adversarial attack methods is defensive distillation (Papernot et al., 2016b), a special form of network distillation, to train a network that becomes almost completely resistant against attacks such as the L-BFGS attack (Szegedy et al., 2014) and the fast gradient sign attack (Goodfellow et al., 2015). However, Carlini & Wagner (2016) recently introduced a novel method for constructing adversarial examples that manages to (very successfully) break many defense methods, including defensive distillation. In fact, the authors find that previous attacks were very fragile and could easily fail to find adversarial examples even when they existed. An experiment on the cross-model adversarial portability (Rozsa et al., 2016) has shown that models with higher accuracies tend to be more robust against adversarial examples, while examples that fool them are more portable to less accurate models.
Even though the existence of adversarial examples has been demonstrated several times on many different classification tasks, the question of why adversarial examples exist in the first place and whether they are sufficiently regular to be detectable, which is studied in this paper, has remained open. Szegedy et al. (2014) speculated that the data-manifold is filled with “pockets” of adversarial inputs that occur with very low probability and thus are almost never observed in the test set. Yet, these pockets are dense and so an adversarial example is found virtually near every test case. The authors further speculated that the high non-linearity of deep networks might be the cause for the existence of these low-probability pockets. Later, Goodfellow et al. (2015) introduced the linear explanation: Given an input and some adversarial noise η (subject to: ||η||∞ < ), the dot product between a weight vector w and an adversarial input xadv = x+ η is given by wTxadv = wTx+ wTη. The adversarial noise η causes a neuron’s activation to grow by wTη. The max-norm constraint on η does not allow for large values in one dimension, but if x and thus η are high-dimensional, many small changes in each dimension of η can accumulate to a large change in a neuron’s activation. The conclusion was that “linear behavior in high-dimensional spaces is sufficient to cause adversarial examples”.
Tanay & Griffin (2016) challenged the linear-explanation hypothesis by constructing classes of images that do not suffer from adversarial examples under a linear classifier. They also point out that if the change in activation wTη grows linearly with the dimensionality of the problem, so does the activation
wTx. Instead of the linear explanation, Tanay et al. provide a different explanation for the existence of adversarial examples, including a strict condition for the non-existence of adversarial inputs, a novel measure for the strength of adversarial examples and a taxonomy of different classes of adversarial inputs. Their main argument is that if a learned class boundary lies close to the data manifold, but the boundary is (slightly) tilted with respect to the manifold1, then adversarial examples can be found by perturbing points from the data manifold towards the classification boundary until the perturbed input crosses the boundary. If the boundary is only slightly tilted, the distance required by the perturbation to cross the decision-boundary is very small, leading to strong adversarial examples that are visually almost imperceptibly close to the data. Tanay et. al further argue that such situations are particularly likely to occur along directions of low variance in the data and thus speculate that adversarial examples can be considered an effect of an over-fitting phenomenon that could be alleviated by proper regularization, though it is completely unclear how to regularize neural networks accordingly.
Recently, Moosavi-Dezfooli et al. (2016a) demonstrated that there even exist universal, imageagnostic perturbations which, when added to all data points, fool deep nets on a large fraction of ImageNet validation images. Moreover, they showed that these universal perturbations are to a certain extent also transferable between different network architectures. While this observation raises interesting questions about geometric properties and correlations of different parts of the decision boundary of deep nets, potential regularities in adversarial perturbations may also help detecting them. However, the existence of universal perturbations does not necessarily imply that the adversarial examples generated by data-dependent adversaries will be regular. Actually, Moosavi-Dezfooli et al. (2016a) show that universal perturbations are not unique and that there even exist many different universal perturbations which have little in common. This paper studies if data-dependent adversarial perturbations can nevertheless be detected reliably and answers this question affirmatively.
3 METHODS
In this section, we introduce the adversarial attacks used in the experiments, propose an approach for detecting adversarial perturbations, introduce a novel adversary that aims at fooling both the classification network and the detector, and propose a training method for the detector that aims at counteracting this novel adversary.
3.1 GENERATING ADVERSARIAL EXAMPLES
Let x be an input image x ∈ R3×width×height, ytrue(x) be a one-hot encoding of the true class of image x, and Jcls(x, y(x)) be the cost function of the classifier (typically cross-entropy). We briefly introduce different adversarial attacks used in the remainder of the paper.
Fast method: One simple approach to compute adversarial examples was described by Goodfellow et al. (2015). The applied perturbation is the direction in image space which yields the highest increase of the linearized cost function under `∞-norm. This can be achieved by performing one step in the direction of the gradient’s sign with step-width ε:
xadv = x+ ε sgn(∇xJcls(x, ytrue(x)))
Here, ε is a hyper-parameter governing the distance between adversarial and original image. As suggested in Kurakin et al. (2016) we also refer to this as the fast method due to its non-iterative and hence fast computation.
Basic Iterative method (`∞ and `2): As an extension, Kurakin et al. (2016) introduced an iterative version of the fast method, by applying it several times with a smaller step size α and clipping all pixels after each iteration to ensure results stay in the ε-neighborhood of the original image:
xadv0 = x, x adv n+1 = Clip ε x { xadvn + α sgn(∇xJcls(xadvn , ytrue(x))) } 1It is easier to imagine a linear decision-boundary - for neural networks this argument must be translated into
a non-linear equivalent of boundary tilting.
Following Kurakin et al. (2016), we refer to this method as the basic iterative method and use α = 1, i.e., we change each pixel maximally by 1. The number of iterations is set to 10. In addition to this method, which is based on the `∞-norm, we propose an analogous method based on the `2-norm: in each step this method moves in the direction of the (normalized) gradient and projects the adversarial examples back on the ε-ball around x (points with `2 distance ε to x) if the `2 distance exceeds ε:
xadv0 = x, x adv n+1 = Project ε x { xadvn + α
∇xJcls(xadvn , ytrue(x)) ||∇xJcls(xadvn , ytrue(x))||2 } DeepFool method: Moosavi-Dezfooli et al. (2016b) introduced the DeepFool adversary which iteratively perturbs an image xadv0 . Therefore, in each step the classifier is linearized around x adv n and the closest class boundary is determined. The minimal step according to the `p distance from xadvn to traverse this class boundary is determined and the resulting point is used as xadvn+1. The algorithm stops once xadvn+1 changes the class of the actual (not linearized) classifier. Arbitrary `p-norms can be used within DeepFool, and here we focus on the `2- and `∞-norm. The technical details can be found in (Moosavi-Dezfooli et al., 2016b). We would like to note that we use the variant of DeepFool presented in the first version of the paper (https://arxiv.org/abs/1511.04599v1) since we found it to be more stable compared to the variant reported in the final version.
3.2 DETECTING ADVERSARIAL EXAMPLES
We augment classification networks by (relatively small) subnetworks, which branch off the main network at some layer and produce an output padv ∈ [0, 1] which is interpreted as the probability of the input being adversarial. We call this subnetwork “adversary detection network” (or “detector” for short) and train it to classify network inputs into being regular examples or examples generated by a specific adversary. For this, we first train the classification networks on the regular (non-adversarial) dataset as usual and subsequently generate adversarial examples for each data point of the train set using one of the methods discussed in Section 3.1. We thus obtain a balanced, binary classification dataset of twice the size of the original dataset consisting of the original data (label zero) and the corresponding adversarial examples (label one). Thereupon, we freeze the weights of the classification network and train the detector such that it minimizes the cross-entropy of padv and the labels. The details of the adversary detection subnetwork and how it is attached to the classification network are specific for datasets and classification networks. Thus, evaluation and discussion of various design choices of the detector network are provided in the respective section of the experimental results.
3.3 DYNAMIC ADVERSARIES AND DETECTORS
In the worst case, an adversary might not only have access to the classification network and its gradient but also to the adversary detector and its gradient2. In this case, the adversary might potentially generate inputs to the network that fool both the classifier (i.e., get classified wrongly) and fool the detector (i.e., look innocuous). In principle, this can be achieved by replacing the cost Jcls(x, ytrue(x)) by (1 − σ)Jcls(x, ytrue(x)) + σJdet(x, 1), where σ ∈ [0, 1] is a hyperparameter and Jdet(x, 1) is the cost (cross-entropy) of the detector for the generated x and the label one, i.e., being adversarial. An adversary maximizing this cost would thus aim at letting the classifier mis-label the input x and making the detectors output padv as small as possible. The parameter σ allows trading off these two objectives. For generating x, we propose the following extension of the basic iterative (`∞) method:
xadv0 = x; x adv n+1 = Clip ε x { xadvn + α [ (1− σ) sgn(∇xJcls(xadvn , ytrue(x))) + σ sgn(∇xJdet(xadvn , 1)) ]} Note that we found a smaller α to be essential for this method to work; more specifically, we use α = 0.25. Since such an adversary can adapt to the detector, we call it a dynamic adversary. To
2We would like to emphasize that is a stronger assumption than granting the adversary access to only the original classifier’s predictions and gradients since the classifier’s predictions need often be presented to a user (and thus also to an adversary). The same is typically not true for the predictions of the adversary detector as they will only be used internally.
counteract dynamic adversaries, we propose dynamic adversary training, a method for hardening detectors against dynamic adversaries. Based on the approach proposed by Goodfellow et al. (2015), instead of precomputing a dataset of adversarial examples, we compute the adversarial examples on-the-fly for each mini-batch and let the adversary modify each data point with probability 0.5. Note that a dynamic adversary will modify a data point differently every time it encounters the data point since it depends on the detector’s gradient and the detector changes over time. We extend this approach to dynamic adversaries by employing a dynamic adversary, whose parameter σ is selected uniform randomly from [0, 1], for generating the adversarial data points during training. By training the detector in this way, we implicitly train it to resist dynamic adversaries for various values of σ. In principle, this approach bears the risk of oscillation and unlearning for σ > 0 since both, the detector and adversary, adapt to each other (i.e., there is no fixed data distribution). In practice, however, we found this approach to converge stably without requiring careful tuning of hyperparameters.
4 EXPERIMENTAL RESULTS
In this section, we present results on the detectability of adversarial perturbations on the CIFAR10 dataset (Krizhevsky, 2009), both for static and dynamic adversaries. Moreover, we investigate whether adversarial perturbations are also detectable in higher-resolution images based on a subset of the ImageNet dataset (Russakovsky et al., 2015).
4.1 CIFAR10
We use a 32-layer Residual Network (He et al., 2016, ResNet) as classifier. The structure of the network is shown in Figure 1. The network has been trained for 100 epochs with stochastic gradient descent and momentum on 45000 data points from the train set. The momentum term was set to 0.9 and the initial learning rate was set to 0.1, reduced to 0.01 after 41 epochs, and further reduced to 0.001 after 61 epochs. After each epoch, the network’s performance on the validation data (the remaining 5000 data points from the train set) was determined. The network with maximal performance on the validation data was used in the subsequent experiments (with all tunable weights being fixed). This network’s accuracy on non-adversarial test data is 91.3%. We attach an adversary detection subnetwork (called “detector” below) to the ResNet. The detector is a convolutional neural network using batch normalization (Ioffe & Szegedy, 2015) and rectified linear units. In the experiments, we investigate different positions where the detector can be attached (see also Figure 1).
4.1.1 STATIC ADVERSARIES
In this subsection, we investigate a static adversary, i.e., an adversary that only has access to the classification network but not to the detector. The detector was trained for 20 epochs on 45000 data points from the train set and their corresponding adversarial examples using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0001 and β1 = 0.99, β2 = 0.999. The remaining 5000 data points from the CIFAR10 train set are used as validation data and used for model selection. The detector was attached to position AD(2) (see Figure 1) except for the DeepFool-based adversaries where the detector was attached to AD(4); see below for a discussion. For the “Fast” and “Iterative” adversaries, the parameter ε from Section 3.1 was chosen from [1, 2, 3, 4] for `∞-based methods and from [20, 40, 60, 80] for `2-based methods; larger values of ε generally result in reduced accuracy of the classifier but increased detectability. For the “Iterative” method with `2-norm, we used α = 20, i.e., in each iteration we make a step of `2 distance 20. Please note that these values of ε are based on assuming a range of [0, 255] per color channel of the input.
Figure 2 (left) compares the detectability3 of different adversaries. In general, points in the lower left of the plot correspond to stronger adversaries because their adversarial examples are harder to detect and at the same time fool the classifier on most of the images. Detecting adversarial examples works surprisingly well given that no differences are perceivable to humans for all shown settings: the detectability is above 80% for all adversaries which decrease classification accuracy below 30% and above 90% for adversaries which decrease classification accuracy below 10%. Comparing the different adversaries, the “Fast” adversary can generally be considered as a weak adversary, the DeepFool based methods as relatively strong adversaries, and the “Iterative” method being somewhere in-between. Moreover, the methods based on the `2-norm are generally slightly stronger than their `∞-norm counter-parts.
Figure 2 (right) compares the detectability of different adversaries for detectors attached at different points to the classification network. ε was chosen minimal under the constraint that the classification accuracy is below 30%. For the “Fast” and “Iterative” adversaries, the attachment position AD(2) works best, i.e., attaching to a middle layer where more abstract features are already extracted but still the full spatial resolution is maintained. For the DeepFool methods, the general pattern is similar except for AD(4), which works best for these adversaries.
Figure 3 illustrates the generalizability of trained detectors for the same adversary with different choices of ε: while a detector trained for large ε does not generalize well to small ε, the other direction works reasonably well. Figure 4 shows the generalizability of detectors trained for one adversary when tested on data from other adversaries (ε was chosen again minimal under the constraint that the
3Detectability refers to the accuracy of the detector. The detectability on the test data is calculated as follows: for every test sample, a corresponding adversarial example is generated. The original and the corresponding adversarial examples form a joint test set (twice the size of the original test set). This test set is shuffled and the detector is evaluated on this dataset. Original and corresponding adversarial example are thus processed independently.
classification accuracy is below 30%): we can see that detectors generalize well between `∞- and `2-norm based variants of the same approach. Moreover, detectors trained on the stronger “Iterative” adversary generalize well to the weaker “Fast” adversary but not vice versa. Detectors trained for the DeepFool-based methods do not generalize well to other adversaries; however, detectors trained for the “Iterative” adversaries generalize relatively well to the DeepFool adversaries.
4.1.2 DYNAMIC ADVERSARIES
In this section, we evaluate the robustness of detector networks to dynamic adversaries (see Section 3.3). For this, we evaluate the detectability of dynamic adversaries for σ ∈ {0.0, 0.1, . . . , 1.0}. We use the same optimizer and detector network as in Section 4.1.1. When evaluating the detectability of dynamic adversaries with σ close to 1, we need to take into account that the adversary might choose to solely focus on fooling the detector, which is trivially achieved by leaving the input unmodified. Thus, we ignore adversarial examples that do not cause a misclassification in the evaluation of the detector and evaluate the detector’s accuracy on regular data versus the successful adversarial examples. Figure 5 shows the results of a dynamic adversary with ε = 1 against a static detector, which was trained to only detect static adversaries, and a dynamic detector, which was explicitly trained to resist dynamic adversaries. As can be seen, the static detector is not robust to dynamic adversaries since for certain values of σ, namely σ = 0.3 and σ = 0.4, the detectability is close to
chance level while the predictive performance of the classifier is severely reduced to less than 30% accuracy. A dynamic detector is considerably more robust and achieves a detectability of more than 70% for any choice of σ.
4.2 10-CLASS IMAGENET
In this section, we report results for static adversaries on a subset of ImageNet consisting of all data from ten randomly selected classes4. The motivation for this section is to investigate whether adversarial perturbations can be detected in higher-resolution images and for other network architectures than residual networks. We limit the experiment to ten classes in order to keep the computational resources required for computing the adversarial examples small and avoid having too similar classes which would oversimplify the task for the adversary. We use a pretrained VGG16 (Simonyan & Zisserman, 2015) as classification network and add a layer before the softmax which selects only the 10 relevant class entries from the logits vector. Based on preliminary experiments, we attach the detector network after the fourth max-pooling layer. The detector network consists of a sequence of five 3x3 convolutions with 196 feature maps each using batch-normalization and rectified linear units, followed by a 1x1 convolution which maps onto the 10 classes, global-average pooling, and a softmax layer. An additional 2x2 max-pooling layer is added after the first convolution. Note that we did not tune the specific details of the detector network; other topologies might perform better than the results reported below. When applicable, we vary ε ∈ [2, 4, 6] for `∞-based methods and ε ∈ [400, 800, 1200] for `2. Moreover, we limit changes of the DeepFool adversaries to an `∞ distance of 6 since the adversary would otherwise sometimes generate distortions which are clearly perceptible. We train the detector for 500 epochs using the Adam optimizer with a learning rate of 0.0001 and β1 = 0.99, β2 = 0.999.
Figure 6 compares the detectability of different static adversaries. All adversaries fail to decrease predictive accuracy of the classifier below the chance level of 0.1 (note that predictive accuracy refers to the accuracy on the 10-class problem not on the full 1000-class problem) for the given values of ε. Nevertheless, detectability is 85% percent or more with the exception of the “Iterative” `2-based adversary with ε = 400. For this adversary, the detector only reaches chance level. Other choices of the detector’s attachment depth, internal structure, or hyperparameters of the optimizer might achieve
4The synsets of the selected classes are: palace; joystick; bee; dugong, Dugong dugon; cardigan; modem; confectionery, confectionary, candy store; valley, vale; Persian cat; stone wall. Classes were selected by randomly drawing 10 ILSVRC2012 Synset-IDs (i.e. integers from [1, 1000]), using the randint function of the python-package numpy after initializing numpy’s random number generator seed with 0. This results in a train set of 10000 images, a validation set of 2848 images, and a test set (from ImageNet’s validation data) of 500 images.
better results; however, this failure case emphasizes that the detector has to detect very subtle patterns and the optimizer might get stuck in bad local optima or plateaus.
Figure 7 illustrates the transferability of the detector between different values of ε. The results are roughly analogous to the results on CIFAR10 in Section 4.1.1: detectors trained for an adversary for a small value of ε work well for the same adversary with larger ε but not vice versa. Note that a detector trained for the “Iterative” `2-based adversary with ε = 1200 can detect the changes of the same adversary with ε = 400 with 78% accuracy; this emphasizes that this adversary is not principally undetectable but that rather the optimization of a detector for this setting is difficult. Figure 8 shows the transferability between adversaries: transferring the detector works well between similar adversaries such as between the two DeepFool adversaries and between the Fast and Iterative adversary based on the `∞ distance. Moreover, detectors trained for DeepFool adversaries work well on all other adversaries. In summary, transferability is not symmetric and typically works best between similar adversaries and from stronger to weaker adversary.
5 DISCUSSION
Why can tiny adversarial perturbations be detected that well? Adopting the boundary tilting perspective of Tanay & Griffin (2016), strong adversarial examples occur in situations in which classification boundaries are tilted against the data manifold such that they lie close and nearly parallel to the data manifold. A detector could (potentially) identify adversarial examples by detecting inputs which are slightly off the data manifold’s center in the direction of a nearby class boundary. Thus, the detector can focus on detecting inputs which move away from the data manifold in a certain direction, namely one of the directions to a nearby class boundary (the detector does not have explicit
knowledge of class boundaries but it might learn about their direction implicitly from the adversarial training data). However, training a detector which captures these directions in a model with small capacity and generalizes to unseen data requires certain regularities in adversarial perturbations. The results of Moosavi-Dezfooli et al. (2016a) suggest that there may exist regularities in the adversarial perturbations since universal perturbations exist. However, these perturbations are not unique and data-dependent adversaries might potentially choose among many different possible perturbations in a non-regular way, which would be hard to detect. Our positive results on detectability suggest that this is not the case for the tested adversaries. Thus, our results are somewhat complementary to Moosavi-Dezfooli et al. (2016a): while they show that universal, image-agnostic perturbations exist, we show that image-dependent perturbations are sufficiently regular to be detectable. Whether a detector generalizes over different adversaries depends mainly on whether the adversaries choose among many different possible perturbations in a consistent way.
Why is the joint classifier/detector system harder to fool? For a static detector, there might be areas which are adversarial to both classifier and detector; however, this will be a (small) subset of the areas which are adversarial to the classifier alone. Nevertheless, results in Section 4.1.2 show that such a static detector can be fooled along with the classifier. However, a dynamic detector is considerably harder to fool: on the one hand, it might further reduce the number of areas which are both adversarial to classifier and detector. On the other hand, the areas which are adversarial to the detector might become increasingly non-regular and difficult to find by gradient descent-based adversaries.
6 CONCLUSION AND OUTLOOK
In this paper, we have shown empirically that adversarial examples can be detected surprisingly well using a detector subnetwork attached to the main classification network. While this does not directly allow classifying adversarial examples correctly, it allows mitigating adversarial attacks against machine learning systems by resorting to fallback solutions, e.g., a face recognition might request human intervention when verifying a person’s identity and detecting a potential adversarial attack. Moreover, being able to detect adversarial perturbations may in the future enable a better understanding of adversarial examples by applying network introspection to the detector network. Furthermore, the gradient propagated back through the detector may be used as a source of regularization of the classifier against adversarial examples. We leave this to future work. Additional future work will be developing stronger adversaries that are harder to detect by adding effective randomization which would make selection of adversarial perturbations less regular. Finally, developing methods for training detectors explicitly such that they can detect many different kinds of attacks reliably at the same time would be essential for safety- and security-related applications.
ACKNOWLEDGMENTS
We would like to thank Michael Herman and Michael Pfeiffer for helpful discussions and their feedback on drafts of this article. Moreover, we would like to thank the developers of Theano (The Theano Development Team, 2016), keras (https://keras.io), and seaborn (http:// seaborn.pydata.org/).

I reviewed the manuscript on December 5th.
Summary:
The authors investigate the phenomenon of adversarial perturbations and ask whether one may build a system to independently detect an adversarial data point -- if one could detect an adversarial example, then might prevent a machine from automatically processing it. Importantly, the authors investigate whether it is possible to build an adversarial detector which is resilient to adversarial examples built against *both* the classifier and the detector. Their results suggest that training a detector in this more difficult setting still yields gains but does not entirely resolve the problem of detecting adversarial examples.
Major comments:
The authors describe a novel approach for dealing with adversarial examples from a security standpoint -- namely, build an independent system to detect the adversary so a human might intervene in those cases.
A potential confound of this approach is that an adversary might respond by constructing adversarial examples to fool *both* the original classifier and the new detector. If that were possible, then this approach is moot since an attacker could always outwit the original system. To their credit, the authors show that building a 'dynamic' detector to detect adversarial examples but also be resilient to an adversary mitigates this potential escalation (worse case from 55% to 70% detection rate). Even though the 'dynamic' detector  demonstrates positive gains, I am concerned about overall scores. Detecting adversarial examples at this rate would not be a reliable security procedure.
My second comment is about 'model transferability'. My definition of 'model transferability' is different then the one used in the paper. My definition means that one constructs an adversarial example on one network and measures how well the adversarial examples attack a second trained model -- where the second model has been trained with different initial conditions. (The author's definition of 'transferability' is based on seeing how well the detector generalizes across training methods). 'Model transferability' (per my definition) is quite important because it measures how general an adversarial example is across all models -- and not specific to a given trained model. Different methods have different levels of 'model transferability' (Kurakin et al, 2016) and I am concerned how well the detector they built would be able to detect adversarial examples across *all models* and not just the trained model in question. In other words, a good detector would be able to detect adversarial examples from any network and not just one particularly trained network. This question seems largely unaddressed in this paper but perhaps I missed some subtle point in their descriptions.
Minor comments:
If there were any points in the bottom-left of the Figure 2 left, then this would be very important to see -- perhaps move the legend to highlight if the area contains no points.
- X-axis label is wrong in Figure 2 right.
Measure the transferability of the detector?
- How is \sigma labeled on Figure 5?
- Whenever an image is constructed to be an 'adversary', has the image actually been tested to see if it is adversarial? In other words, does the adversarial image actually result in a misclassification by the original network?