Despite the long history of research on recommender systems, current approaches still face a number of challenges in practice, e.g. the difficulties in handling new items, the high diversity of user interests, and the noisiness and sparsity of observations. Many of such difficulties stem from the lack of expressive power to capture the complex relations between items and users. This paper presents a new method to tackle this problem, called Collaborative Deep Embedding. In this method, a pair of dual networks, one for encoding items and the other for users, are jointly trained in a collaborative fashion. Particularly, both networks produce embeddings at multiple aligned levels, which, when combined together, can accurately predict the matching between items and users. Compared to existing methods, the proposed one not only provides greater expressive power to capture complex matching relations, but also generalizes better to unseen items or users. On multiple real-world datasets, this method outperforms the state of the art.
Despite the long history of research on recommender systems, current approaches still face a number of challenges in practice, e.g. the difficulties in handling new items, the high diversity of user interests, and the noisiness and sparsity of observations. Many of such difficulties stem from the lack of expressive power to capture the complex relations between items and users. This paper presents a new method to tackle this problem, called Collaborative Deep Embedding. In this method, a pair of dual networks, one for encoding items and the other for users, are jointly trained in a collaborative fashion. Particularly, both networks produce embeddings at multiple aligned levels, which, when combined together, can accurately predict the matching between items and users. Compared to existing methods, the proposed one not only provides greater expressive power to capture complex matching relations, but also generalizes better to unseen items or users. On multiple real-world datasets, this method outperforms the state of the art.
1 INTRODUCTION
What do consumers really want? – this is a question to which everyone wishes to have an answer. Over the past decade, the unprecedented growth of web services and online commercial platforms such as Amazon, Netflix, and Spotify, gives rise to a vast amount of business data, which contain valuable information about the customers. However, “data don’t speak for themselves”. To accurately predict what the customers want, one needs not only the data, but also an effective means to extract useful messages therefrom.
There has been extensive study on recommender systems. Existing methods roughly fall into two categories, namely content-based filtering (Pazzani & Billsus, 2007) and collaborative filtering (Mnih & Salakhutdinov, 2008; Hu et al., 2008; Yu et al., 2009). The former focuses on extracting relevant features from the content, while the latter attempts to exploit the common interest among groups of users. In recent efforts, hybrid methods (Agarwal & Chen, 2009; Van den Oord et al., 2013) that combine both aspects have also been developed.
Whereas remarkable progress has been made on this topic, the state of the art remains far from satisfactory. The key challenges lie in several aspects. First, there is a large semantic gap between the true cause of a matching and what we observe from the data. For example, what usually attracts a book consumer is the implied emotion that one has to feel between the lines instead of the occurrences of certain words. It is difficult for classical techniques to extract such deep meanings from the observations. Second, the cold-start issue, namely making predictions for unseen items or users, has not been well addressed. Many collaborative filtering methods rely on the factorization of the matching matrix. Such methods implicitly assume that all the users and items are known in advance, and thus are difficult to be applied in real-world applications, especially online services.
The success of deep learning brings new inspiration to this task. In a number of areas, including image classification (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and natural language understanding (Socher et al., 2011), deep learning techniques have substantially pushed forward the state of the art. The power of deep networks in capturing complex variations and bridging semantic gaps has been repeatedly shown in previous study. However, deep models were primarily used for classification or regression, e.g. translating images to sentences. How deep networks can be used to model cross-domain relations remains an open question.
In this work, we aim to explore deep neural networks for learning the matching relations across two domains, with our focus placed on the matching between items and users. Specifically, we propose a new framework called Collaborative Deep Embedding, which comprises a pair of dual networks, one for encoding items and the other for users. Each network contains multiple embedding layers that are aligned with their dual counterparts of the other network. Predictions can then be made by coupling these embeddings. Note that unlike a conventional network, the dual networks are trained on two streams of data. In this paper, we devise an algorithm that can jointly train both networks using dual mini-batches. Compared to previous methods, this method not only narrows the semantic gap through a deep modeling architecture, but also provides a natural way to generalize – new items and new users can be encoded by the trained networks, just like those present in the training stage.
On a number of real world tasks, the proposed method yields significant improvement over the current state-of-the-art. It is worth stressing that whereas our focus is on the matching between items and users, Collaborative Deep Embedding is a generic methodology, which can be readily extended to model other kinds of cross-domain relations.
2 RELATED WORK
Existing methods for recommendation roughly fall into two categories: content-based methods (Pazzani & Billsus, 2007) and collaborative filtering (CF) (Mnih & Salakhutdinov, 2008; Hu et al., 2008; Yu et al., 2009). Specifically, content-based methods rely primarily on feature representation of the content, in which recommendations are often made based on feature similarity (Slaney et al., 2008). Following this, there are also attempts to incorporate additional information, such as meta-data of users, to further improve the performance (McFee et al., 2012). Instead, collaborative filtering exploits the interaction between users and items. A common approach to CF is to derive latent factors of both users and items through matrix factorization, and measure the degree of matching by their inner products. Previous study (Ricci et al., 2011) showed that CF methods tend to have higher recommendation accuracy than content-based methods, as they directly target the recommendation task. However, practical use of CF is often limited by the cold start problem. It is difficult to recommend items without a sufficient amount of use history. Issues like this motivated hybrid methods (Agarwal & Chen, 2009; Van den Oord et al., 2013) that combine both aspects of information, which have showed encouraging improvement. Our exploration is also along this line.
Despite the progress on both family of methods, the practical performance of state-of-the-art still leaves a lot to be desired. This, to a large extent, is due to the lack of capability of capturing complex variations in interaction patterns. Recently, deep learning (Bengio, 2009) emerges as an important technique in machine learning. In a number of successful stories (Krizhevsky et al., 2012; Hinton et al., 2012; Socher et al., 2011), deep models have demonstrated remarkable representation power in capturing complex patterns. This power has been exploited by some recent work for recommendation. Van den Oord et al. (2013) applies deep learning for music recommendation. It uses the latent item vector learned by CF as ground truth to train a deep network for extracting content features, obtaining considerable performance gain. However the latent vectors for known users and items are not improved. Wang & Wang (2014) proposed an extension to this method, which concatenates both the CF features and the deep features, resulting in slight improvement.
Wang & Blei (2011) showed that CF and topic modeling, when combined, can benefit each other. Inspired by this, Wang et al. (2015) proposed Collaborative Deep Learning (CDL), which incorporates CF and deep feature learning with a combined objective function. This work represents the latest advances in recommendation methods. Yet, its performance is still limited by several issues, e.g. the difficulties in balancing diversified objectives and the lack of effective methods for user encoding. An important aspect that distinguishes our work from CDL and other previous methods is that it encodes both items and users through a pair of deep networks that are jointly trained, which substantially
enhance the representation power on both sides. Moreover, the objective function of our learning framework directly targets the recommendation accuracy, which also leads to better performance.
3 COLLABORATIVE DEEP EMBEDDING
At the heart of a recommender system is matching model, namely, a model that can predict whether a given item matches the interest of a given user. Generally, this can be formalized as below. Suppose there are m users and n items, respectively indexed by i and j. Items are usually associated with inherent features, e.g. the descriptions or contents. Here, we use xj to denote the observed features of the j-th item. However, inherent information for users is generally very limited and often irrelevant. Hence, in most cases, users are primarily characterized by their history, i.e. the items they have purchased or rated. Specifically, the user history can be partly captured by a matching matrix R ∈ {0, 1}m×n, where R(i, j) = 1 indicates that the i-th user purchased the j-th item and gave a positive rating. Note that R is often an incomplete reflection of the user interest – it is not uncommon that a user does not purchase or rate an item that he/she likes.
3.1 DUAL EMBEDDING
To motivate our approach, we begin with a brief revisit of collaborative filtering (CF), which is widely adopted in practical recommender systems. The basic idea of CF is to derive vector representations for both users and items by factorizing the matching matrix R. A representative formulation in this family is the Weighted Matrix Factorization (WMF) (Hu et al., 2008), which adopts an objective function as below: ∑
i ∑ j cij(Rij − uTi vj)2 + λu ∑ i ‖ui‖22 + λv ∑ j ‖vj‖22. (1)
Here, ui and vj denote the vector representations of the i-th user and the j-th item, cij the confidence coefficient of an observed entry, and λu, λv the regularization coefficients. Underlying such methods lies a common assumption, namely, all users and items must be known a priori. As a result, they will face fundamental difficulties when handling new items and new users.
Encoding Networks. In this work, we aim to move beyond this limitation by exploring an alternative approach. Instead of pursuing the embeddings of a given set of items and users, our approach jointly learns a pair of encoding networks, respectively for items and users. Compared to CF, the key advantage of this approach is that it is generalizable by nature. When new items or new users come, their vector embeddings can be readily derived using the learned encoders.
Generally, the items can be encoded based on their own inherent features, using, for example, an auto-encoder. The key question here, however, is how to encode users, which, as mentioned, have no inherent features. Again, we revisit conventional CF methods such as WMF and find that in these methods, the user representations can be expressed as:
ui = argmin u ∑ j cij‖Rij − uTi vj‖2 + λu ∑ i ‖ui‖2 = ( VCiV T + λuI )−1 Vri. (2)
Here, V = [v1, . . . ,vn] is a matrix comprised of all item embeddings, each column for one; ri is the i-th row of R treated as a column vector, which represents the history of the i-th user; and Ci = diag(ci1, . . . , cin) captures the confidence weights.
The analysis above reveals that ui is a linear transform of ri as ui = Wuri, where the transform matrix Wu depends on the item embeddings V. This motivates our idea of user encoding, that is, to use a deep neural network instead the linear transform above, as
ui = g(ri;Wu), (3)
where g denotes a nonlinear transform based on a deep network with parameters Wu. As we will show in our experiments, by drawing on the expressive power of deep neural networks, the proposed way of user encoding can substantially improve the prediction accuracy.
Overall Formulation. By coupling an item-network denoted by f(xj ;Wv) and a user-network g as introduced above, we can predict the matching of any given pair of user and item based on the inner product of their embeddings, as 〈f(x;Wv), g(r;Wu)〉. The inputs to these networks include x, the inherent feature of the given item, and r, the history of the given user on a set of reference items. With both encoding networks, we formulate the learning objective as follows:
min Wu,Wv ∑ i ∑ j cij‖Rij − 〈f(xj ;Wv), g(ri;Wu)〉‖2. (4)
Here, X = [x1, . . . ,xn] denotes the input features of all reference items. This formulation differs from previous ones in two key aspects: (1) Both users and items are encoded using deep neural networks. The learning objective above encourages the cooperation of both networks such that the coupling of both sides yield the highest accuracy. Hence, the user-network parameters Wu depends on the item embeddings V, and likewise for the item-network. (2) The learning task is to estimate the parameters of the encoding networks. Once the encoding networks are learned, they encode users and items in a uniform way, no matter whether they are seen during training. In other words, new users and new items are no longer second-class citizens – they are encoded in exactly the same way as those in the training set.
Comparison with CDL. The Collaborative Deep Learning (CDL) recently proposed by Wang et al. (2015) was another attempt to tackle the cold-start issue. This method leverages the item features by aligning the item encoder with the embeddings resulted from matrix factorization. In particular, the objective function is given as follows:∑ ij cij(Rij−uTi vj)2+λv ∑ j ‖vj−fe(x̃j ,θ)‖2+λn ∑ j ‖x̃j−fr(x̃j ,θ)‖2+λu ∑ i ‖ui‖2+r(θ). (5) Here, a Stacked Denoising Autoencoder (SDAE) (Vincent et al., 2010) with parameter θ is used to encode the items, based on {x̃j}, noisy versions of their features. Compared to our formulation, CDL has several limitations: (1) The objective is to balance the SDAE reconstruction error and the matching accuracy, which does not necessarily lead to improved recommendation. Tuning this balance also turns out to be tricky. (2) Only items are encoded, while the representations of the users are still obtained by matrix factorization. As a result, its expressive power in capturing user interest remains limited. (3) There are inconsistencies between known items and new ones – the embedding of known items is resulted from a tradeoff between the matching accuracy and the fidelity to SDAE features, while the embedding of new items are purely based on SDAE encoding.
3.2 NETWORK ARCHITECTURE DESIGNS
Our model consists of two networks, namely the item-network f and the user-network g. We went through a progressive procedure in designing their architectures, obtaining three different designs, from basic design, multi-level design, to multi-level branching design. Each new design was motivated by the observation of certain limitations in the previous version.
The basic design, as shown in Figure 1 (a) adopts the multilayer perceptron as the basic architecture, using tanh as the nonlinear activation function between layers1. The top layer of the item-network produces a vector f(xj ;Wv) for each item; while that of the user-network produces a dual vector g(ri;Wu) for each user. During training, the loss layer takes their inner products and compares them with the ground-truth R(i, j).
Each layer in these networks generates a vector representation. We observe that representations from different layers are complementary. Representations from lower layers tend to be closer to the inputs and preserve more information; while those from higher layers focus on deeper semantics. The representations from these levels have their respective values, as different users tend to focus on different aspects of an item. Following this intuition, we reach a multi-level design, as shown in Figure 1 (b). In this design, dot products between dual embeddings at corresponding levels are aggregated to produce the final prediction.
There is an issue of the multi-level design – the output of each intermediate layer actually plays two roles. On one hand, it is the input to the next layer for further abstraction; on the other hand, it also serves as a facet to be matched with the other side. These two roles require different properties of the representations. Particularly, for the former role, the representation needs to preserve more information for higher-level abstraction; while for the latter, those parts related to the current level of matching need to be emphasized. To address this issue, we design a multi-level branching architecture, as shown in Figure 1 (c). In this design, a matching branch is introduced to transform the representation at each level to a form that is more suitable for matching. This can also be considered as learning an alternative metric to measure the matchness between the embeddings. As we will show in our experiments, this design can considerably improve the prediction accuracy.
4 TRAINING WITH DUAL MINI-BATCHES
A distinctive aspect of our training algorithm is the use of dual mini-batches. Specifically, in each iteration, Bv items and Bu users are selected. In addition to the item features and user histories, the corresponding part of the matching matrix R will also be loaded and fed to the network. Here, the two batch sizes Bv and Bu can be different, and they should be chosen according to the sparsity of the matching matrix R, such that each dual mini-batch can cover both positive and zero ratings.
During the backward pass, the loss layer that compares the predictions with the ground-truth matchings will produce two sets of gradients, respectively for items and users. These gradients are then back-propagated along respective networks. Note that when the multi-level designs (both with and without branching) are used, each intermediate layer will receive gradients from two sources – those from the upper layers and those from the dual network (via the dot-product layer). Hence, the training of one network would impact that of the other.
The entire training procedure consists of two stages: pre-training and optimization. In the pre-training stage, we initialize the item-network with unsupervised training (Vincent et al., 2010) and the usernetwork randomly. The unsupervised training of the item-network allows it to capture the feature statistics. Then both networks will be jointly refined in a layer-by-layer fashion. Particularly, we first tune the one-level networks, taking the dot products of their outputs as the predictions. Subsequently, we stack the second layers on top and refine them in a similar way. Empirically, we found that this layer-wise refinement scheme provides better initialization. In the optimization stage, we adopt the SGD algorithm with momentum and use the dual mini-batch scheme presented above. In this stage, the training is conducted in epochs. Each epoch, through multiple iterations, traverses the whole matching matrix R without repetition. The order of choosing mini-batches is arbitrary and will be shuffled at the beginning of each epoch. Additional tricks such as dropout and batch normalization are employed to further improve the performance.
1The choice of tanh as the activation function is based on empirical comparison.
5 EXPERIMENTS
We tested our method on three real-world datasets with different kinds of items and matching relations:
1. CiteULike, constructed by Wang & Blei (2011), provides a list of researchers and the papers that they interested. Each paper comes with a text document that comprises both the title and the abstract. In total, it contains 5, 551 researchers (as users) and 16, 980 papers (as items) with 0.22% density. The task is to predict the papers that a researcher would like.
2. MovieLens+Posters is constructed based on the MovieLens 20M Dataset (Harper & Konstan, 2016), which provides about 20M user ratings on movies. For each movie, we collect a movie poster from TMDb and extract a visual feature therefrom using a convolutional neural network (Szegedy et al., 2016) as the item feature. Removing all those movies without posters and the users with fewer than 10 ratings, we obtain a dataset that contains 76, 531 users and 14, 101 items with 0.24% density. In this dataset, all 5 ratings are considered as positive matchings.
3. Ciao is organized by Tang et al. (2012) from a product review site, where each product comes with a series of reviews. The reviews for each product are concatenated to serve as the item content. We removed those items with less than 5 rated users and the users with less than 10 ratings. This results in a dataset with 4, 663 users and 12, 083 items with 0.25% density. All ratings with 40 or above (the rating ranges from 0 to 50) are regarded as positive matchings.
5.1 EVALUATION
The performance of a recommender system can be assessed from different perspective. In this paper, we follow Wang & Blei (2011) and perform the evaluation from the retrieval perspective. Specifically, a fraction of rating entries are omitted in the training phase, and the algorithms being tested will be used to predict those entries. As pointed out by Wang & Blei (2011), as the ratings are implicit feedback (Hu et al., 2008) – some positive matchings are not reflected in the ratings, recall is more suitable than precision in measuring the performance. In particular, we use Recall@M averaged over all users as the performance metric. Here, for a certain user, Recall@M is defined as follows:
recall@M = the number of items a user likes in top M recommendations
the total number of items the user likes .
In our experiments, the value of M varies from 50 to 200.
Following Wang & Blei (2011), we consider two tasks, in-matrix prediction and out-matrix prediction. Specifically, we divide all users into two disjoint parts, known and unknown, by the ratio of 9 to 1. The in-matrix prediction task only considers known items. For this task, all rating entries are split into three disjoint sets: training, validation and testing, by the ratio 3 : 1 : 1. It is ensured that all items in the validation and testing sets have appeared in the training stage (just that part of their ratings were omitted). The out-matrix prediction task is to make predictions for the items that are completely unseen in the training phase. This task is to test the performance of generalization and the capability of handling the cold-start issue.
5.2 COMPARISON WITH OTHER METHODS
We compared our method, which we refer to as DualNet with two representative methods in previous work: (1) Weighted Matrix Factorization (WMF) (Hu et al., 2008), a representative method for for collaborative filtering (CF), and (2) Collaborative deep learning (CDL) (Wang et al., 2015), a hybrid method that combines deep encoding of the items and CF, which represents the latest advances in recommendation techniques.
On each dataset, we chose the design parameters for each method via grid search. The parameter combinations that attain best performance on the validation set are used. For our DualNet method, we adopt a three-level branching configuration, where the embedding dimensions of each network, from bottom to top, are set to 200, 200, 50. For WMF, the latent dimension is set to 300 on CDL and 450 on other datasets. For CDL, the best performance is attained when the structure of SDAE is configured to be (2000, 1000, 300), with drop out ratio 0.1. Other design parameters of CDL are set as a = 1.0, b = 0.01, lu = 1, lv = 10, ln = 1000, lw = 0.0005.
Note that on CiteULike, there are two ways to split the data. One is the scheme in (Wang et al., 2015), and the other is the scheme in (Wang & Blei, 2011), which is the one presented in the previous section. Note that in the former scheme, a fixed number of ratings from each user are selected for training. This may result in some testing items being missed in the training set. To provide a complete comparison with prior work, we use both schemes in our experiments, which are respectively denoted as CiteULike1 and CiteULike2.
Table 1 compares the performance of WML, CDL, and DualNet on all three datasets (four data splitting settings). From the results, we observed: (1) Our proposed DualNet method outperforms both WML and CDL on all datasets. On certain data sets, the performance gains are substantial. For example, on MovieLens, we obtained average recalls at 44.95%, 59.15%, and 72.56% respectively when M = 50, 100, 200. Comparing what CDL achieves (38.11%, 49, 73%, and 61.00%), the relative gains are around 18%. On other data sets, the gains are also considerable. (2) The performance gains vary significantly across different datasets, as they are closely related to the relevance of the item features. Particularly, when the item features are pertinent to the user interest, we may see remarkable improvement when those features are incorporated; otherwise, the performance gains would be relatively smaller.
5.3 DETAILED STUDY
We conducted additional experiments on CiteULike to further study the proposed algorithm. In this study, we investigate the performance of out-matrix prediction, the impact of various modeling choices, e.g. multi-level branching, as well as the influence of training tactics.
Out-matrix prediction. As mentioned, the out-matrix prediction task is to examine an algorithm’s capability of handling new items, i.e. those unseen in the training stage. For this task, we compared CDL and DualNet on the CiteULike dataset. WML is not included here as it is not able to handle new items. Table 2 shows the results. It can be clearly seen that DualNet outperforms CDL by a notable margin. For example, Recall@50 increases from 32.18% to 47.51% – the relative gain is 47.6%, a very remarkable improvement. The strong generalization performance as demonstrated here is, to a large extent, ascribed to our basic formulation, where the encoding networks uniformly encode both known and new items.
Multi-level branching. We compared three different designs presented in Section 3: basic design, multi-level design, and multi-level branching design. From the results shown in Table 3, we can observe limited improvement of the multi-level design over the basic one. More significant performance
gains are observed when the branching design is introduced. This shows that the branches contribute a lot to the overall performance.
Noise injection. Sometimes we noticed overfitting during training i.e. the validation performance gets worse while the training loss is decreasing. To tackle this issue, we inject noises to the inputs, i.e. setting a fraction of input entries to zeros. Generally, we observed that noise injection has little effect for Recall@M on in-matrix predictions when M < 30. However, it can considerably increase the recall for largeM value or out-matrix predictions. Particularly, on CiteULike, it increases in-matrix Recall@300 from 67.3% to 71.2%, and out-matrix Recall@50 from 38.6% to 47.5%.
Unsuccessful Tactics. Finally, we show some tactics that we have tried and found to be not working. (1) Replacing the weighted Euclidean loss with logistic loss would lead to substantial degradation of the performance (sometimes by up to 20%). Also, when using logistic loss, we observed severe overfitting. Rendle et al. (2009) proposed Bayesian Personalized Recommendation (BPR) which directly targets on ranking. We tested this on CiteULike with parameters tuned to obtain the optimal performance. Our experimental results showed that its performance is similar to WMF. Particularly, the Recall@50, 100, 200 for BPR are respectively 39.11%, 49.16%, 59.96%, while those for WMF are 40.45%, 50.25%, 59.95%.
(2) Motivated by the observation that positive ratings are sparse, we tried a scheme that ignores a fraction of dual mini-batches that correspond to all zero ratings, with an aim to speed up the training. Whereas this can reduces the time needed to run an epoch, it takes significantly more epochs to reach the same level of performance. As a result, the overall runtime is even longer.
6 CONCLUSIONS AND FUTURE WORK
This paper presented a new method for predicting the interactions between users and items, called Collaborative Deep Embedding. This method uses dual networks to encode users and items respectively. The user-network and item-network are trained jointly, in a collaborative manner, based on two streams of data. We obtained considerable performance gains over the state-of-the-art consistently on three large datasets. The proposed method also demonstrated superior generalization performance (on out-matrix predictions). This improvement, from our perspective, is ascribed to three important reasons: (1) the expressive power of deep models for capturing the rich variations in user interests, (2) the collaborative training process that encourages closely coupled embeddings, and (3) an objective function that directly targets the prediction accuracy.
We consider this work as a significant step that brings the power of deep models to relational modeling. However, the space of deep relational modeling remains wide open – lots of questions remain yet to be answered. In future, we plan to investigate more sophisticated network architectures, and extend the proposed methodology to applications that involve more than two domains.

The responses to the pre-review questions are not strong; especially w.r.t. the question about dataset density and why the dataset had to be subsampled, the authors responded that subsampling is common in recommender systems work, including the papers cited. This isn't a particularly strong justification of why subsampling is a good idea, and in particular doesn't answer the question of "how would the results look without subsampling," which I think is a question that could easily have been answered directly.
Especially given that the goal of dealing with the cold-start issue is so heavily emphasized in the paper, in seems odd to sample the data to reduce sparsity.
Other than that, the pre-review questions seem to have been answered satisfactorily.
The contribution of the paper is to propose user and item embedding methods, as a means of learning complex non-linear interactions between users and items. This is fairly similar to recent work on deep RS, though the network formulation has some differences.
Overall this is an reasonably put together paper that makes a contribution in an important area, though there are still some shortcomings that should be addressed, namely:
1) The evaluation is unusual. Recall@M is the only result reported, though this is not usually an evaluation seen in recommender systems research. At the very least other performance measures (rmse or AUC) should be reported for completeness, even if the results are not strong
2) Given that the contribution is fairly simple (i.e., the "standard" recommender systems task, but with a new model) it's a shame that unusual data samples have to be taken. This should be a case where it's possible to report results against competing methods using *exactly* the same data they used, and exactly the same error measure, for the fairest comparison possible.
Without the above it's hard to tell how much the performance improvements are really due to the method being better, versus the choice of datasets and the choice of loss functions.


Due to deep cascades of nonlinear units, deep neural networks (DNNs) can automatically learn non-local generalization priors from data and have achieved high performance in various applications. However, such properties have also opened a door for adversaries to generate the so-called adversarial examples to fool DNNs. Specifically, adversaries can inject small perturbations to the input data and therefore decrease the performance of deep neural networks significantly. Even worse, these adversarial examples have the transferability to attack a black-box model based on finite queries without knowledge of the target model. Therefore, we aim to empirically compare different defensive strategies against various adversary models and analyze the cross-model efficiency for these robust learners. We conclude that the adversarial retraining framework also has the transferability, which can defend adversarial examples without requiring prior knowledge of the adversary models. We compare the general adversarial retraining framework with the state-of-the-art robust deep neural networks, such as distillation, autoencoder stacked with classifier (AEC), and our improved version, IAEC, to evaluate their robustness as well as the vulnerability in terms of the distortion required to mislead the learner. Our experimental results show that the adversarial retraining framework can defend most of the adversarial examples notably and consistently without adding additional vulnerabilities or performance penalty to the original model.
1 INTRODUCTION
Despite the success of deep neural networks (DNNs) in diverse areas, ranging from image recognition and machine translation to autonomous driving, its vulnerabilities have been exploited in the adversarial environments. Evasion attacks against such deep learning systems have recently received considerable attention. It has been shown that with small magnitude of noise added, the original instance can easily be misclassified by the otherwise accurate deep neural networks (Goodfellow et al., 2014; Papernot et al., 2016c; Nguyen et al., 2015; Szegedy et al., 2013). Such instances are also called adversarial examples.
Given the strong evasion properties of these adversarial examples, some works have been proposed to test and investigate the robustness of the deep neural networks against the adversarial examples (Goodfellow et al., 2014; Kurakin et al., 2016; Huang et al., 2015; Gu & Rigazio, 2014; Jin et al., 2015). However, most of the existing works only evaluate the robustness of the proposed defense strategies over adversarial examples generated using a single attack method, or several similar methods. Meanwhile, since the evaluated adversary models, i.e., adversarial example generation methods, vary among different works that study the effectiveness of defense strategies, it remains a question how to make a comparison among different defense strategies.
In this paper, we focus on providing thorough analysis for different algorithmic strategic defensive learners against various adversary models considering their robustness against adversarial examples,
efficiency for cross-model learning process, resilience against additional attacks, and the vulnerabilities of these learners. Here by “cross-model test”, we mean to apply one adversarial model to generate adversarial examples, while test them on the learner trained with instances generated from different adversarial models. High “cross-model test” accuracy indicates higher robustness for learner. In addition, we propose to test the “additional attacks” in a repeated game setting to estimate learner based on against further attacks. A nice symmetry analysis for both the adversary and learner is provided through these analyses. We show that the general adversarial retraining framework performs significantly robust compared with the state-of-the-art defensive algorithms. For example, even for the black box attack, which is considered hard to defend, as long as there is a way to generate these adversarial evasion examples, the robust adversarial retraining framework can always improve the learning ability without knowing the actual adversary model. To our best knowledge, this work is the first to provide comprehensive analysis for different adversarial models and possible defensive solutions.
In summary, we made the following contributions:
1. Evaluate the robustness of the general robust adversarial retraining framework (RAD) with the state-of-the-art AEC, Distillation, and the improved AEC, against different adversary models;
2. Propose an improved AutoEncoder stacked with Classifier (IAEC);
3. Compare the cross-model learning efficiency of different defensive methods and demonstrate the ability to defend against black-box attacks;
4. Demonstrate the robustness of the retraining framework RAD, AEC, IAEC, and Distillation, against new attacks by attacking these robust learners repeatedly;
5. Analyze the vulnerabilities induced by different defensive strategies/models based on their tolerance of the malicious distortions required to mislead the classifier.
We illustrate the applicability and efficiency of different defensive strategies against various stateof-the-art adversary models based on both MNIST and CIFAR-10 datasets.
2 RELATED WORK
Efforts have been made to understand adversarial examples. Goodfellow et al. (2014) pointed out that the adversarial examples actually make use of the linear nature of the DNNs based on the observation of their generalization across architectures and training sets. Tabacof & Valle (2015) analyzed the adversarial image space and showed that adversarial images appear in large regions in the pixel space. Papernot et al. (2016c) studied the limitation of adversarial evasion examples and showed that some instances are more difficult to manipulate than the others. Sabour et al. (2015) demonstrated that the attacker can change classification to an arbitrary class by malicious manipulations. The reverse engineering problem has been proposed in Vorobeychik & Li (2014), and it theoretically proved that the black-box attack is possible and also showed one could learn a sufficiently similar classifier from queries both theoretically and empirically. Similarly, even without knowing exactly the learning algorithm, several black-box attacks have been proposed targeting DNNs, which demonstrates the transferability of such adversarial examples (Papernot et al., 2016a;b).
Some training methods have been proposed to improve the robustness of deep neural networks. Jan et al. (2002) has proposed to explore the perturbed regions and apply ensemble method to enhance the robustness of classification. Zheng et al. have proposed to stabilize the state-of-the-art Inception architecture against different distortions, and it focuses on general random noise or distortions, such as compression, rescaling and cropping on images Zheng et al. (2016). Miyato et al. (2015) have proposed to apply the local distribution smoothness for statistical model to promote the smoothness of the model distribution and conduct the virtual adversarial training to enhance the performance of deep neural networks. However, all these works did not test on the adversarial examples and still had a long way to perform robustly against these real adversarial instances.
While the existence of adversarial examples is attracting more and more attention, some defense strategies have been proposed to defend against such adversarial examples. In Goodfellow et al. (2014), Goodfellow et al. proposed to train the network with an adversarial objective function based
on fast gradient sign method: J̃(θ, x, y) = αJ(θ, x, y) + (1 − α)J(θ, x + ∇xJ(θ, x, y), y). In a concurrent and independent work, Kurakin et al. provided an adversarial training method for a large scale dataset, i.e., ImageNet dataset Kurakin et al. (2016). However, they only used fast gradient sign-based methods to generate adversarial examples for both training and evaluation, which fails to consider the generality of defensive strategy. Huang et al. Huang et al. (2015) proposed an alternative method for adversarial training by considering an empirically stronger adversary. In their work, suppose r? is the optimal adversarial perturbation for an instance (x, y), instead of adding (x + r?, y) into the training dataset, they used “pseudo-samples” for training controlled by a hyperparameter c, which represents the magnitude of perturbation, i.e., (x + c r ?
||r?|| , y). Several autoencoder structures Vincent et al. (2008) have been proposed against the adversarial examples by reconstructing the original images ahead of classification Gu & Rigazio (2014). Jin et al. have proposed a feedforward CNN structure to improve the robustness in the presence of adversarial noise, which is restricted to the specific type of models in Jin et al. (2015). However, the focus of these researches perform too aggressively on designing robust learning algorithms against arbitrary small perturbations (e.g., noise) neglecting the properties of actual adversarial evasion models. Therefore, studying various adversarial models and building resilient learners accordingly is important. Here we will provide comparisons for defensive algorithms facing different adversarial models to provide insights and encourage devising more efficient learner.
3 PROBLEM
To understand the phenomenon of adversarial examples in deep neural networks, we aim to analyze potential defending methods against different adversary models from various perspectives, such as the robustness of the learner itself, the cross-model generalization ability, the resilience against additional attacks, and the vulnerabilities in terms of the required distortion to attack the robust learner again. Let X ⊆ Rn represent the feature space, with n the number of features. For every instance xi ∈ X , which is drawn from certain distribution xi ∼ D, there is a corresponding label yi ∈ Y to comprise the data pair (xi, yi), where xij denotes the jth feature of xi. In the adversarial environments, adversary would like to accomplish the goal of evading the classifier. To formalize, suppose that M ⊆ Y is a set of labels which an adversary wishes to attack, and let z(m) be the target label for each m ∈ M . For example, for autonomous driving, potential adversaries may aim to manipulate a stop sign or a dead-end warning sign, to a lamppost, a tree, or an advertisement sign, to cause accidents. Since such perturbations on images towards deep neural networks are often imperceptible to human eyes, it can cause serious vulnerabilities when deploying the DNNs in real adversarial environments. The defender’s goal is to learn a classifier with parameters w, gw : xi → Y , using a training data set of labeled instance T = {(x1, y1), ..., (xm, ym)}. Here, we focus on deep neural networks representing the function gw(·). Therefore, the learner’s objective is to minimize the following general loss function:
min w L(w;A) = ∑ i:yi∈Y\M l(gw(xi), yi) + ∑ i:yi∈M l(gw(A(w, xi), yi) + α‖w‖pp, (1)
where l(·) can be arbitrary loss function and A represents the adversary model. The adversarial risk function in Equation 1 is general: it can be any adversary model oracle, A, which is used to generate the adversarial evasion instances. Traditionally, this adversarial oracle may capture evasion attack models based on minimizing evasion cost (Lowd & Meek, 2005; Li & Vorobeychik, 2014; Biggio et al., 2014), or based on actual attacker evasion behavior obtained from experimental data (Ke et al., 2016). More formally, we will discuss the potential adversary models for deep neural networks and the possible defensive models for the learner in detail below.
3.1 ADVERSARY MODEL
To mislead deep neural networks, various methods have been proposed to generate the adversarial examples. We mainly discuss three state-of-the-art adversary models A here for further evaluation. Fast Gradient Sign. Based on the linear view of adversarial examples, a fast way of generating these adversarial examples were proposed in Goodfellow et al. (2014). Suppose xi is the original feature vector, based on adversary modelA(fgs), we have xi′ = xi+η, where η represents the perturbation
added for the original instance. Therefore, the dot product between the weighted parameter vector w and an adversarial example xi′ becomes:
wTxi ′ = wTxi + w T η.
Let J(w, xi, yi) be the cost used to train the neural network. By linearizing the cost function around the current value of w, an optimal max-norm constrained perturbation is generated as
η = sign(OxJ(w, xi, yi)),
where the adversary can vary to generate adversarial examples with different attacking abilities for different deep neural networks.
Coordinate Greedy. Another more general adversary model A(cg) is the local search framework Coordinate Greedy (cg) proposed in Li et al. (2016) for approximating the optimal adversarial instance. As an illustration, we focus on binary classification, and assume that gw(x) = sign(f(x)) for some continuous function f , which in this case would be represented by a deep neural network.
The coordinate greedy approach is quite general, but we consider a specific adversary objective in which the adversary here tries to balance between two considerations: 1) appear as benign as possible to the classifier, and 2) minimize the cost of modification of the original instance (e.g., minimally manipulate the image). Note that it is also natural to assume that the attacker obtains no value from a manipulation to the original feature vector if the result is still classified as malicious. Therefore, an adversary aiming to transform an instance xi into an adversarial example xi′ is solving the following optimization problem:
min xi′∈X
min{0, f(xi′)}+ c(xi′, xi), (2)
where c(xi′, xi) is the cost function of modifying from xi to xi′. Here c(xi′, xi) ≥ 0, c(xi′, xi) = 0 iff xi′ = xi, and the cost function c is strictly increasing in ‖xi′ − xi‖2 and strictly convex in xi′. Because Problem 2 is non-convex, so the objective of adversary can be formed to minimize an upper bound:
min xi′
Q(xi ′) ≡ f(xi′) + c(xi′, xi). (3)
So the high-level idea of cg is to iteratively choose a feature, and greedily update this feature according to the partial derivatives of the attacker’s objective as 3 to evade the classifier. Below, we take the exponential cost function c(xi′, xi) = exp ( λ( ∑ j(xij ′ − xij)2 + 1)1/2 )
as an example to estimate the modification cost, which is also quite natural: options become exponentially less desirable to an attacker as they are more distant from their ideal attack. Then we take the following partial derivative to update the adversary’s objective until the convergence.
∂Q(xi ′)
∂xij = ∂f(xi
′)
∂xij + ∂c(xi
′, xi)
∂xij = ∂f(xi
′)
∂xij +
λc(xi ′, xi)(xij ′ − xij) ( ∑ j(xij ′ − xij)2 + 1)1/2 ,
To avoid the algorithm converges only to a locally optimal solution, random restarts strategy is applied to randomly select the starting points in the feature space. As long as a global optimum has a basin of attraction with positive Lebesgue measure, or the feature space is finite, this process will asymptotically converge to a globally optimal solution with enough random restarts.
Adam. Another adversary model A(adam) applies the stochastic gradient-based optimization algorithm Adam Kingma & Ba (2014) to generate adversarial examples. Specifically, the adversary uses Adam to solve the same optimization problem as in Equation 3.
3.2 DEFENDER MODEL
Given the possible adversary models, several defensive strategies have been proposed focusing on different perspectives. Basically, the learner tries to integrate the prior knowledge of either the adversary model or the data distribution with the classification process. Here we consider different defensive strategies given the adversary model and form the interaction as a Stackelberg game. We will also consider the repeated game setting in section 4.4.
Adversarial Retraining framework (RAD). A systematic defensive approach based on adversarial retraining (RAD) has been proposed in Li et al. (2016). At the high level, RAD starts with the original training data and iteratively updating the learner with adversarial instances that evade the previously computed classifier until the convergence. It has been proved that the algorithm will terminate and the lower bound of the empirical loss of RAD is also provided. The important part for RAD is to select the adversarial retraining instances. In practice, it is hard to exactly estimate the adversary model as well as the parameters used within their model. Therefore, the generalization ability of RAD across different adversary models is quite important. Surprisingly, RAD generalizes quite well among various adversary models without requiring to know the exact attacker strategy. We will present the cross-model analysis for RAD in details in section 4.3.
AutoEncoder stacked with Classifier (AEC). One of the recent and efficient defensive method is the AutoEncoder stacked with a classifier to initialize deep architectures proposed in Gu & Rigazio (2014). To assess the structure of the adversarial noise, an autoencoder on mapping adversarial examples back to the original data samples is trained and stacked with the classifier. We train the AutoEncoder with different adversarial algorithms, including the fast gradient sign method (fgs), the coordinate greedy (cg) method, as well as Adam.
Improved AutoEncoder stacked with Classifier (IAEC). Since the baseline AEC cannot perform very well by only mapping the adversarial images back to the original image, we apply an improved AutoEncoder stacked with classifier (IAEC) defensive method. As AutoEncoder itself can not ensure that adversarial examples are denoised, we add a cross-entropy regularizer term as the loss function to help ensure that the output of AutoEncoder is classified correctly. Let yi be the one-hot representation of ground truth label of an input instance xi, then our loss function becomes:
J(xi) = ‖s(xi)− xi′‖+H(yi, f(xi)),
where s(xi) represents the mapping result of xi by the AutoEncoder, and the cross-entropy function H(yi, f(xi)) = − ∑ xi yi log f(xi).
Distillation. Considering the fact that the knowledge extracted during training, which is in the form of probability vectors, and transferred in smaller networks to maintain accuracy comparable with those of larger networks can also be beneficial to improving generalization capabilities of deep neural networks outside of their training dataset, a defensive strategy against the adversarial examples has been proposed in Papernot et al. (2015). This defensive strategy transfers the knowledge contained in probability vectors through the distillation training step, then applies these probabilities in the next training step instead of using the original hard labels, and therefore enhances its resilience to perturbations. This defensive model is independent of the adversary models and we will evaluate its robustness and vulnerabilities in details in section 4.
4 EXPERIMENTAL ANALYSIS
In this section, we empirically compare the adversarial retraining framework RAD with other stateof-the-art baseline methods Distillation Papernot et al. (2015), AutoEncoder stacked with Classifier (AEC) Gu & Rigazio (2014) and our improved AutoEncoder stacked with Classifier (IAEC) against various adversary models based on both MNIST and CIFAR-10 datasets.
Basically, we first analyze the robustness of RAD and Distillation, which performs the best against adversarial examples currently, by comparing the classification results before and after applying the adversarial retraining technique based on both MNIST and CIFAR-10 datasets. Then we estimate the cross-model classification robustness for RAD, AEC, the improved IAEC, and Distillation. Precisely, during the cross-model evaluation, we allow the attacker to generate the adversarial examples with different adversarial algorithms, while the defender has no clue about what adversarial algorithm is used. Therefore, we are able to evaluate the resilience of the “black-box” defensive strategies without requiring to know the actual adversary model.
Besides, we allow the attacker to attack these robustly enhanced learners and we compare the resilience of the RAD with the baseline defensive models and show that with retraining instances generated by adam, the RAD is almost unassailable for attacks based on the fast gradient sign method, which is promising to design universal defensive algorithms based on RAD.
Additionally, another perspective to measure the robustness of the learners is to evaluate how much noise is needed to make the learner misclassify an otherwise correct instance. As pointed out by Gu & Rigazio (2014), even a learner can be demonstrated to perform robustly against certain adversarial examples, it may become more vulnerable in the sense of being attacked by adding much smaller magnitude of adversarial noise. This means increasing the noticeability of the smallest adversarial noise for each example becomes the key to solve the adversarial examples problem. Therefore, we compare the malicious distortion required to attack each model, aiming to evaluate the vulnerability of different learners. The distortion is measured by d(xi′, xi) = 1n √∑ (xi′ − xi)2, where xi′ = A(β, xi) representing the adversarial manipulated instance based on arbitrary adversary model A.
4.1 EXPERIMENTAL SETUP
In our experiments, we focus on binary classification, and the adversary tries to modify a malicious instance (classified as +1) to evade the classifier and be classified as benign (-1). On MNIST, we select digit “4” as the malicious (positive) class, and “7” as the benign (negative) class. On CIFAR10, we use “Airplane” as the malicious class, and “Cat” as the benign class. We use LeNet-5 LeCun et al. (1998) to perform the binary classification, and all classifiers used to evaluate the efficiency of different adversary models and defender models are based on this model architecture. All input pixel values are normalized into [−0.5, 0.5]. With respect to adversary models, during the evaluation, all of them modify the malicious instances in the original testset to evade the classifier, and keep the benign instances untouched. Meanwhile, for iterative attack methods evaluated in our experiments, i.e., cg and adam, according to our experiments, actually we can find adversarial examples with small modification cost using any λ, even when setting λ = 0, i.e., not considering the cost function c(xi′, xi) for optimization. Therefore, we set λ = 0 for all experiments using these two attack methods.
With respect to defensive models, for RAD, we only add adversarial examples generated on original malicious instances into the dataset for retraining, since the goal of adversary is trying to fool a classifier to label a malicious instance as benign, which follows the framework proposed in Li et al. (2016). As for AEC and IAEC, we use the same autoencoder architecture for removing adversarial noises proposed in Gu & Rigazio (2014), i.e., a three-hidden-layer autoencoder (784-256-128-256- 784 neurons). We train the autoencoder to map adversarial examples generated on original malicious instances to the original images, and as suggested in Gu & Rigazio (2014), we also train the autoencoder to map original data back to itself. Both AEC and IAEC stack the autoencoder with a LeNet-5 classifier.
4.2 ROBUSTNESS ANALYSIS FOR DEFENSIVE LEARNERS
To evaluate the robustness and efficiency of the adversarial retraining framework and other defensive learners, we generate adversarial examples based on the the coordinate gradient algorithm (cg), adam, and the fast gradient sign algorithm (fgs ) with the size of perturbation = 0.1 ∼ 0.5 ( Goodfellow et al. (2014)), respectively. Figure 1 shows the analysis of recall for the traditional LeNet-5 and the robust RAD classifiers on MNIST. The test error of LeNet-5 on the original dataset is 0.045%. It is obvious that after the adversarial retraining process based on RAD, the classifiers perform nearly optimal. It is interesting to observe that with the of fgs increases, the adversarial examples generated by fgs can attack the original LeNet-5 more efficiently.
Figure 2 presents the comparisons of recall for the original LeNet-5 and the adversarial retraining framework on CIFAR-10. It shows that the adversarial retraining framework works robustly against different adversarial example generation methods. Note the test error of LeNet-5 on the original dataset is 5.5%. From the results of recall, we can see that almost all the “generated” adversarial instances are correctly classified by the retraining framework. Additionally, sometimes the test error of RAD is even smaller than that of the original model LeNet-5 based on the uncontaminated (no adversary) data. This means, with the adversarial robust retraining process, some “blind-spots” in the input space volume can be filled out without decreasing the performance on the normal test data. Moreover, surprisingly, with the increase of , the fast gradient sign method works worse for generating adversarial examples against LeNet-5, which is different for MNIST. This is actually caused by the properties of the fast gradient sign method itself. By following the gradient, the generated instance can be trapped into sub-optimal and therefore fail to converge to the global optima,
so different step size can affect their final convergence. Therefore, by comparing with the results of MNIST, we can see learners on CIFAR-10 is easier to be trapped by the sub-optima and larger values can lead the learner to be trapped into these points with higher probability. On the other hand, no matter how much the strength of adversarial ability is affected by different parameters, the adversarial retraining framework works robustly by almost identifying all the manipulated instances correctly on different datasets consistently.
4.3 CROSS-MODEL ANALYSIS FOR DIFFERENT DEFENSIVE LEARNERS
Aiming to defend a more broad class of attacks, here we assume the the learner has no clue about which adversarial algorithm the attacker uses to generate the adversarial instances. Therefore, the defender can perform robustly as the “black-box” learner against arbitrary adversaries. Here we use different attack algorithms to generate the adversarial examples, and the retraining instances for RAD are also generated across various adversary models to evaluate the learners’ generalization ability. We also compare the results with the state-of-the-art Distillation, AEC and our improved IAEC algorithm based on different adversarial models. Here the AEC is trained on the adam model, which offers the best classification results. The IAEC is also trained corresponding to different adversary models to compare the cross-model learning ability with RAD. Table 1 shows the test error comparisons for these cross-model learners. “No adversary” presents the test error of different learners on the clean data. Basically, the adversarial retraining framework performs consistently better than AEC, IAEC, and Distillation on all different adversarial examples in terms of the classification error. This conclusion is independent of what models are used to generate adversarial retraining instances for RAD. Based on the results, the adversarial retraining framework has the potential to be applied to defend against any arbitrary attacks without requiring to know the exact adversary model. Based on the classification error results for our improved IAEC in Table 1, it is obvious that the IAEC with the same adam adversary model works much more robust than AEC. This means the proposed IAEC is much more robust compared with the original AEC by adding the cross-entropy regularization. Additionally, we also evaluate the cross-model classification error for IAEC to test its generalization ability. Table 1 shows that the IAEC can also defend against different adversarial examples without requiring to know the exact adversary model.
Similarly, we show the classification error comparison results of RAD across different adversary models in Table 2 compared with Distillation. As CIFAR-10 images are more complex, the error rates for adversarial retraining framework get larger compared with that on MNIST. However, overall the classification error for the retraining framework on different adversarial examples are below 13% with zero knowledge of the adversary model, while the classification error on normal data is around 6%. Therefore, even on CIFAR-10 dataset, the adversarial retraining framework is still promising to perform the “black-box” defending resiliently against various attacks. Additionally, the distillation with T = 1 and T = 100 both encounter higher test error than RAD, even the distillation method performs more robustly when T = 100 than T = 1.
4.4 ROBUSTNESS AGAINST ADDITIONAL ATTACKS
In order to test the robustness of the learner against the repeated attacks, where the attacker can again conduct attacks on the robust learners, here we evaluate how the robust learner behaves given additional attacks based on different adversary models. Table 3 presents the test error rate comparison when the attacker generates adversarial examples to attack the robust RAD learner, IAEC, and Distillation on MNIST. It is shown that the coordinate greedy (cg) and adam are somehow efficient to attack RAD, while the fast gradient sign methods fail to attack the robust RAD. So if the RAD is retrained with instances generated by arbitrary adversary models, it can be resilient against adversarial examples produced by the fast gradient sign method with various values. This means the RAD can confer robustness to single-step attack methods but not the iterative ones. However, adversaries based on cg and adam can still find the vulnerabilities to attack the model. Compared with the performance of the adversarial retraining framework (RAD) against these “repeated attacks”, the IAEC encounters much higher classification error when being attacked. This indicates that the adversarial retraining framework can not only enhance the resilience of the original learner (LeNet-5), but also perform robustly against the additional attacks compared with the IAEC.
Similarly, Table 4 presents the test error for attacking different robust learners with various adversary models on CIFAR-10. RAD again produces lower test error compared with Distillation (T = 1,
T = 100) given diverse adversarial attacking strategies. What is worth to mention is that these robust learners all perform accurately on the normal dataset without adversarial manipulation, which offers more potentials for the robust learners.
4.5 VULNERABILITY OF THE DEFENSIVE LEARNERS
Given the fact that the attacker can attack the learning model continuously, here we are concerned with how vulnerable the robust models become in terms of the amount of distortion needed to add to mislead the learner. We compare the average distortion for attacking the LeNet-5, RAD, IAEC, and Distillation to evaluate their robustness. As mentioned by Gu & Rigazio (2014), AEC demands smaller distortion to attack, which means AEC is quite fragile, and we also gain the similar observation and confirm that attacking the original LeNet-5 model requires larger magnitude of noise than AEC. Thus, we focus on the improved IAEC.
In the Table 5 we present the demanded distortion to maliciously attack the RAD, the IAEC, and Distillation on MNIST. Note that the fast gradient sign method here is a one-step method, which will stop after computing one gradient to find the optimal perturbation of a linear approximation of the cost or model, so it cannot guarantee to find the evasion instance xi′ and we do not consider its distortion. so here we only consider cg and adam to generate distortions. We use RAD(.) to represent the adversarial retraining framework retrained with arbitrary adversarial instances since they all require the same amount of distortion to be attacked given their similar network structures. From Table 5 RAD requires the same distortion as attacking the original LeNet-5 model. However, the distortion needed for attacking the IAEC is substantially smaller than that for attacking the original models. From this perspective, the IAEC becomes more vulnerable compared with the original model even though it can be resilient against the adversarial examples. Similar for Distillation, smaller distortion is demanded to attack the robust learner, which means more vulnerabilities are introduced by the robust Distillation. On the contrary, the adversarial retraining framework RAD can perform robustly against various diverse adversarial attacks without increasing the vulnerability penalty.
Figure 3 shows the results of adding the corresponding adversarial noise to generate the misclassification for LeNet-5 model by different adversarial algorithms qualitatively. It shows that by using fast gradient sign method with = 0.5, the original image is almost distorted. This indicates different adversary models have different attacking strengths, so taking the stronger adversary model into
account may have a chance to defend the weaker adversaries, which makes the universal defensive model promising.
Similarly, Table 6 lists the amount of distortion needed to fool the original learner based on CIFAR10. It is shown that both the RAD and Distillation need exactly the same amount of distortion with the original LeNet-5 model, which means these robust learners do not increase the vulnerability of the original model.
The visual attacking results by injecting malicious noise are shown in Figure 4. It is clear that fgs with = 0.5 can distort the original images the most compared with other adversary algorithms. Surprisingly, all the retraining framework based on different retraining instances only get the classification error lower than 3.0%.
5 CONCLUSION
To understand the adversarial examples better, as well as the potential adversary models and corresponding defensive learners, we conduct extensive experiments to evaluate properties of different defensive strategies. We point out that RAD works the best among all the defensive strategies against different adversary models, including one-step and iterative ones, in terms of the classification test
error. The adversarial retraining framework, RAD, also generalizes well for the cross-model evaluation compared with AEC, IAEC, and Distillation. Moreover, both RAD and Distillation do not introduce additional vulnerability penalty to the original models, while still increase the robustness. So in the future work, to generalize the robust learner across different adversary models, one direction could be to generate retraining instances based on diverse adversarial algorithms to cover as much as possible the “blind-spots” within the input space. In addition, we will dynamically optimize the choice of adversary model and the quantity of retraining instances according to the robustness requirements of a specific learner. Therefore, the tradeoff between robustness and accuracy on the normal data can be balanced based on the specific resilience demand of the learner.

This paper performs a series of experiments to systematically evaluate the robustness of several defense methods, including RAD, AEC and its improved version etc.. It provides interesting observations. Overall, RAD and distillation have the best performances, but none of the methods can really resist the 'additional' attack from cg or adam. Since it is an experimental paper, my main concern is about its clarity. See the comments below for details.
Pros:
1. This paper provides a good comparison of the performances for the selected methods.
2. Section 3.3 (the 'additional' attack) is a interesting investigation. Although the final result about the defense methods is negative, its results are still inspiring.
3. Overall, this paper provides interesting and inspiring experimental results about the selected methods.
Cons:
1. There are several other methods in the literature that are missing from the paper. For example the defense methods and the attack methods in the papers [1,2].
2. Although a long list of experimental results are provided in the paper, many details are skipped. For example, details of the experiments that generate the results in Table 5.
3. Without further explanations and analyses about the experimental results, the contribution of the paper seems limited.
4. This paper proposed an improved version of the AEC algorithm. But its experimental results seems not promising.
Minor comments:
Page 3: Equation (3) is also non-convex. So the non-convexity of Equation (2) should not be the motivation of Equation (3).
[1]