Trusting SVM for Piecewise Linear CNNs
We present a novel layerwise optimization algorithm for the learning objective of Piecewise-Linear Convolutional Neural Networks (PL-CNNs), a large class of convolutional neural networks. Specifically, PL-CNNs employ piecewise linear non-linearities such as the commonly used ReLU and max-pool, and an SVM classifier as the final layer. The key observation of our approach is that the problem corresponding to the parameter estimation of a layer can be formulated as a difference-of-convex (DC) program, which happens to be a latent structured SVM. We optimize the DC program using the concave-convex procedure, which requires us to iteratively solve a structured SVM problem. This allows to design an optimization algorithm with an optimal learning rate that does not require any tuning. Using the MNIST, CIFAR and ImageNet data sets, we show that our approach always improves over the state of the art variants of backpropagation and scales to large data and large network settings.
1 Introduction
The backpropagation algorithm is commonly employed to estimate the parameters of a convolutional neural network (CNN) using a supervised training data set (Rumelhart et al., 1986). Part of the appeal of backpropagation comes from the fact that it is applicable to a wide variety of networks, namely those that have (sub-)differentiable non-linearities and employ a (sub-)differentiable learning objective. However, the generality of backpropagation comes at the cost of a high sensitivity to its hyperparameters such as the learning rate and momentum. Standard line-search algorithms cannot be used on the primal objective function in this setting, as (i) there may not exist a step-size guaranteeing a monotonic decrease because of the use of sub-gradients, and (ii) even in the smooth case, each function evaluation requires a forward pass over the entire data set without any update, making the approach computationally unfeasible. Choosing the learning rate thus remains an open issue, with the state-of-the-art algorithms suggesting adaptive learning rates (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015). In addition, techniques such as batch normalization (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014) have been introduced to respectively reduce the sensitivity to the learning rate and to prevent from overfitting.
With this work, we open a different line of inquiry, namely, is it possible to design more robust optimization algorithms for special but useful classes of CNNs? To this end, we focus on the networks that are commonly used in computer vision. Specifically, we consider CNNs with convolutional and dense layers that apply a set of piecewise linear (PL) non-linear operations to obtain a discriminative representation of an input image. While this assumption may sound restrictive at first, we show that commonly used non-linear operations such as ReLU and max-pool fall under the category of PL functions. The representation obtained in this way is used to classify the image via a multi-class SVM, which forms the final layer of the network. We refer to this class of networks as PL-CNN.
We design a novel, principled algorithm to optimize the learning objective of a PL-CNN. Our algorithm is a layerwise method, that is, it iteratively updates the parameters of one layer while keeping the other layers fixed. For this work, we use a simple schedule over the
layers, namely, repeated passes from the output layer to the input one. However, it may be possible to further improve the accuracy and efficiency of our algorithm by designing more sophisticated scheduling strategies. The key observation of our approach is that the parameter estimation of one layer of PL-CNN can be formulated as a difference-of-convex (DC) program that can be viewed as a latent structured SVM problem (Yu & Joachims, 2009). This allows us to solve the DC program using the concave-convex procedure (CCCP) (Yuille & Rangarajan, 2002). Each iteration of CCCP requires us to solve a convex structured SVM problem. To this end, we use the powerful block-coordinate Frank-Wolfe (BCFW) algorithm (Lacoste-Julien et al., 2013), which solves the dual of the convex program iteratively by computing the conditional gradients corresponding to a subset of training samples. In order to further improve BCFW for PL-CNNs, we extend it in three important ways. First, we introduce a trust-region term that allows us to initialize the BCFW algorithm using the current estimate of the layer parameters. Second, we reduce the memory requirement of BCFW by an order of magnitude, via an efficient representation of the feature vectors corresponding to the dense layers. Third, we show that, empirically, the number of constraints of the structural SVM problem can be reduced substantially without any loss in accuracy, which allows us to significantly reduce its time complexity.
Compared to backpropagation (Rumelhart et al., 1986) or its variants (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015), our algorithm offers three advantages. First, the CCCP algorithm provides a monotonic decrease in the learning objective at each layer. Since layerwise optimization itself can be viewed as a block-coordinate method, our algorithm guarantees a monotonic decrease of the overall objective function after each layer’s parameters have been updated. Second, since the dual of the SVM problem is a smooth convex quadratic program, each step of the BCFW algorithm (in the inner iteration of the CCCP) provides a monotonic increase in its dual objective. Third, since the only step-size required in our approach comes while solving the SVM dual, we can use the optimal step-size that is computed analytically during each iteration of BCFW (Lacoste-Julien et al., 2013). In other words, our algorithm has no learning rate, initial or not, that requires tuning.
Using standard network architectures and publicly available data sets, we show that our algorithm provides a boost over the state of the art variants of backpropagation for learning PL-CNNs and we demonstrate scalability of the method.
2 Related Work
While some of the early successful approaches for the optimization of deep neural networks relied on greedy layer-wise training (Hinton et al., 2006; Bengio et al., 2007), most currently used methods are variants of backpropagation (Rumelhart et al., 1986) with adaptive learning rates, as discussed in the introduction.
At every iteration, backpropagation performs a forward pass and a backward pass on the network, and updates the parameters of each layer by stochastic or mini-batch gradient descent. This makes the choice of the learning rate critical for efficient optimization. Duchi et al. (2011) have proposed the Adagrad convex solver, which adapts the learning rate for every direction and takes into account past updates. Adagrad changes the learning rate to favor steps in gradient directions that have not been observed frequently in past updates. When applied to the non-convex CNN optimization problem, Adagrad may converge prematurely due to a rapid decrease in the learning rate (Goodfellow et al., 2016). In order to prevent this behavior, the Adadelta algorithm (Zeiler, 2012) makes the decay of the learning rate slower. It is worth noting that this fix is empirical, and to the best of our knowledge, provides no theoretical guarantees. Kingma & Ba (2015) propose a different scheme for the learning rate, called Adam, which uses an online estimation of the first and second moments of the gradients to provide centered and normalized updates. However all these methods still require the tuning of the initial learning rate to perform well.
Second-order and natural gradient optimization methods have also been a subject of attention. The focus in this line of work has been to come up with appropriate approximations to make the updates cheaper. Martens & Sutskever (2012) suggested a Hessian-free second order optimization using finite differences to approximate the Hessian and conjugate gradient to
compute the update. Martens & Grosse (2015) derive an approximation of the Fisher matrix inverse, which provides a more efficient method for natural gradient descent. Ollivier (2013) explore a set of Riemannian methods based on natural gradient descent and quasi-Newton methods to guarantee reparametrization invariance of the problem. Desjardins et al. (2015) demonstrate a scaled up natural gradient descent method by training on the ImageNet data set (Russakovsky et al., 2015). Though providing more informative updates and solid theoretical support than SGD-based approaches, these methods do not take into account the structure of the problem offered by the commonly used non-linear operations.
Our work is also related to some of the recent developments in optimization for deep learning. For example, Taylor et al. (2016) use ADMM for massive distribution of computation in a layer-wise fashion, and in particular their method will yield closed-form updates for any PLCNN. Lee et al. (2015) propose to use targets instead of gradients to propagate information through the network, which could help to extend our algorithm. Zhang et al. (2016) derive a convex relaxation for the learning objective for a restricted class of CNNs, which also relies on solving an approximate convex problem. In (Amos et al., 2016), the authors identify convex problems for the inference task, when the neural network is a convex function of some of its inputs.
With a more theoretical approach, Goel et al. (2016) propose an algorithm to learn shallow ReLU nets with guarantees of time convergence and generalization error. Heinemann et al. (2016) show that a subclass of neural networks can be modeled as an improper kernel, which then reduces the learning problem to a simple SVM with the constructed kernel.
More generally, we believe that our hitherto unknown observation regarding the relationship between PL-CNNs and latent SVMs can (i) allow the progress made in one field to be transferred to the other and (ii) help design a new generation of principled algorithms for deep learning optimization.
3 Piecewise Linear Convolutional Neural Networks
A piecewise linear convolutional neural network (PL-CNN) consists of a series of convolutional layers, followed by a series of dense layers, which provides a concise representation of an input image. Each layer of the network performs two operations: a linear transformation (that is, a convolution or a matrix multiplication), followed by a piecewise linear non-linear operation such as ReLU or max-pool. The resulting representation of the image is used for classification via an SVM. In the remainder of this section, we provide a formal description of PL-CNN.
Piecewise Linear Functions. A piecewise linear (PL) function f(u) is a function of the following form (Melzer, 1986):
f(u) = max i∈[m] {a>i u} −max j∈[n] {b>j u}, (1)
where [m] = {1, · · · ,m}, and [n] = {1, · · · , n}. Each of the two maxima above is a convex function, therefore such a function f is not generally convex, but it is rather a difference of two convex functions. Importantly, many commonly used non-linear operations such as ReLU or max-pool are PL functions of their input. For example, ReLU corresponds to the function R(v) = max{v, 0} where v is a scalar. Similarly, max-pool for a D-dimensional vector u corresponds to M(u) = maxi∈[D]{e>i u}, where ei is a vector whose i-th element is 1 and all other elements are 0. Given a value of u, we say that (i∗, j∗) is the activation of the PL function at u if i∗ = argmaxi∈[m]{a>i u} and j∗ = argmaxj∈[n]{b>j u}.
PL-CNN Parameters. We denote the parameters of an L layer PL-CNN by W = {W l; l ∈ [L]}. In other words, the parameters of the l-th layer is defined as W l. The CNN defines a composite function, that is, the output zl−1 of layer l− 1 is the input to the layer l. Given the input zl−1 to layer l, the output is computed as zl = σl(W l · zl−1), where “·” is either a convolution or a matrix multiplication, and σl is a PL non-linear function, such as ReLU or max-pool. The input to the first layer is an image x, that is, z0 = x. We denote
the input to the final layer by zL = Φ(x;W) ∈ RD. In other words, given an image x, the convolutional and dense layers of a PL-CNN provide a D-dimensional representation of x to the final classification layer. The final layer of a PL-CNN is a C class SVM W svm, which specifies one parameter W svmy ∈ RD for each class y ∈ Y.
Prediction. Given an image x, a PL-CNN predicts its class using the following rule:
y∗ = argmax y∈Y W svmy Φ(x;W). (2)
In other words, the dot product of the D-dimensional representation of x with the SVM parameter for a class y provides the score for the class. The desired prediction is obtained by maximizing the score over all possible classes.
Learning Objective. Given a training data set D = {(xi, yi), i ∈ [N ]}, where xi is the input image and yi is its ground-truth class, we wish to estimate the parameters W ∪W svm of the PL-CNN. To this end, we minimize a regularized upper bound on the empirical risk. The risk of a prediction y∗i given the ground-truth yi is measured with a user-specified loss function ∆(y∗i , yi). For example, the standard 0− 1 loss has a value of 0 for a correct prediction and 1 for an incorrect prediction. Formally, the parameters of a PL-CNN are estimated using the following learning objective:
min W,W svm
λ
2 ∑ l∈[L]∪{svm} ‖W l‖2F + 1 N N∑ i=1 max ȳi∈Y ( ∆(ȳi, yi) + ( W svmȳi −W svm yi )T Φ(xi;W) ) . (3)
The hyperparameter λ denotes the relative weight of the regularization compared to the upper bound of the empirical risk. Note that, due to the presence of piecewise linear non-linearities, the representation Φ(·;W) (and hence, the above objective) is highly non-convex in the PL-CNN parameters.
4 Parameter Estimation for PL-CNN
In order to enable layerwise optimization of PL-CNNs, we show that parameter estimation of a layer can be formulated as a difference-of-convex (DC) program (subsection 4.1). This allows us to use the concave-convex procedure, which solves a series of convex optimization problems (subsection 4.2). We show that each convex problem closely resembles a structured SVM objective, which can be addressed by the powerful block-coordinate Frank-Wolfe (BCFW) algorithm. We extend BCFW to improve its initialization, time complexity and memory requirements, thereby enabling its use in learning PL-CNNs (subsection 4.3). For the sake of clarity, we only provide sketches of the proofs for those propositions that are necessary for understanding the paper. The detailed proofs of the remaining propositions are provided in the Appendix.
4.1 Layerwise Optimization as a DC Program
Given the values of the parameters for the convolutional and the dense layers (that is, W), the learning objective (3) is the standard SVM problem in parameters W svm. In other words, it is a convex optimization problem with several efficient solvers (Tsochantaridis et al., 2004; Joachims et al., 2009; Shalev-Shwartz et al., 2009), including the BCFW algorithm (LacosteJulien et al., 2013). Hence, the optimization of the final layer is a computationally easy problem. In contrast, the optimization of the parameters of a convolutional or a dense layer l does not result in a convex program. In general, this problem can be arbitrarily hard to solve. However, in the case of PL-CNN, we show that the problem can be formulated as a specific type of DC program, which enables efficient optimization via the iterative use of BCFW. The key property that enables our approach is the following proposition that shows that the composition of PL functions is also a PL function.
Proposition 1. Consider PL functions g : Rm → R and gi : Rn → R, for all i ∈ [m]. Define a function f : Rn → R as f(u) = g([g1(u), g2(u), · · · , gm(u)]>). Then f is also a PL function (proof in Appendix A).
Using the above proposition, we can reformulate the problem of optimizing the parameters of one layer of the network as a DC program. Specifically, the following proposition shows that the problem can be formulated as a latent structured SVM objective (Yu & Joachims, 2009).
Proposition 2. The learning objective of a PL-CNN with respect to the parameters of the l-th layer can be specified as follows:
min W l
λ 2 ‖W l‖2F + 1 N N∑ i=1
max hi∈H ȳi∈Y
( ∆(ȳi, yi) + (W l)>Ψ(xi, ȳi,hi) ) − max
hi∈H
( (W l)>Ψ(xi, yi,hi) ) ,
(4)
for an appropriate choice of the latent space H and joint feature vectors Ψ(x, y,h) of the input x, the output y and the latent variables h. In other words, parameter estimation for the l-th layer corresponds to minimizing the sum of its Frobenius norm plus a PL function for each training sample.
Sketch of the Proof. For a given image x with the ground-truth class y, consider the input to the layer l, which we denote by zl−1. Since all the layers except the l-th one are fixed, the input zl−1 is a constant vector, which only depends on the image x (that is, its value does not depend on the variables W l). In other words, we can write zl−1 = ϕ(x).
Given the input zl−1, all the elements of the output of the l-th layer, denoted by zl, are a PL function of W l since the layer performs a linear transformation of zl−1 according to the parameters W l, followed by an application of PL operations such as ReLU or max-pool. The vector zl is then fed to the (l + 1)-th layer. The output zl+1 of the (l + 1)-th layer is a vector whose elements are PL functions of zl. Therefore, by proposition (1), the elements of zl+1 are a PL function of W l. By applying the same argument until we reach the layer L, we can conclude that the representation Φ(x;W) is a PL function of W l. Next, consider the upper bound of the empirical risk, which is specified as follows:
max ȳ∈Y
( ∆(ȳ, y) + ( W svmȳ −W svmy )T Φ(x;W) ) . (5)
Once again, since W svm is fixed, the above upper bound can be interpreted as a PL function of Φ(x;W), and thus, by proposition (1), the upper bound is a PL function of W l. It only remains to observe that the learning objective (3) also contains the Frobenius norm of W l. Thus, it follows that the estimation of the parameters of layer l can be reformulated as minimizing the sum of its Frobenius norm and the PL upper bound of the empirical risk over all training samples, as shown in problem (4). Note that we have ignored the constants corresponding to the Frobenius norm of the parameters of all the fixed layers. This constitutes an existential proof of Proposition 2. In the next paragraph, we give an intuition about the feature vectors Ψ(xi, ȳi,hi) and the latent space H.
Feature Vectors & Latent Space. The exact form of the joint feature vectors depends on the explicit DC decomposition of the objective function. In Appendix B, we detail the practical computations and give an example: we construct two interleaved neural networks whose outputs define the convex and concave parts of the DC objective function. Given the explicit DC objective function, the feature vectors are given by a subgradient and can therefore be obtained by automatic differentiation.
We now give an intuition of what the latent space H represents. Consider an input image x and a corresponding latent variable h ∈ H. The latent variable can be viewed as a set of variables hk, k ∈ {l + 1, · · · , L}. In other words, each subset hk of the latent variable corresponds to one of the layers of the network that follow the layer l. Intuitively, hk represents the choice of activation at layer k when going through the PL activation: for each neuron j of layer k, hkj takes value i if and only if the i-th piece of the piecewise linear activation is selected. For instance, i is the index of the selected input in the case of a max-pooling unit.
Note that the latent space only depends on the layers that follow the current layer being optimized. This is due to the fact that the input zl−1 to the l-th layer is a constant vector
that does not depend on the value of W l. However, the activations of all subsequent layers following the l-th one depend on the value of the parameters W l. As a consequence, the greater the number of following layers, the greater the size of the latent space, and this growth happens to be exponential. However, as will be seen shortly, it is still possible to efficiently optimize problem (4) for all the layers of the network despite this exponential increase.
4.2 Concave-Convex Procedure
The optimization problem (4) is a DC program in the parameters W l. This follows from the fact that the upper bound of the empirical risk is a PL function, and can therefore be expressed as the difference of two convex PL functions (Melzer, 1986). Furthermore, the Frobenius norm of W l is also a convex function of W l. This observation allows us to obtain an approximate solution of problem (4) using the iterative concave-convex procedure (CCCP) (Yuille & Rangarajan, 2002).
Algorithm 1 describes the main steps of CCCP. In step 3, we impute the best value of the latent variable corresponding to the ground-truth class yi for each training sample. This imputation corresponds to the linearization step of the CCCP. The selected latent variable corresponds to a choice of activations at each non-linear layer of the network, and therefore defines a path of activations to the ground truth. Next, in step 4, we update the parameters by solving a convex optimization problem. This convex problem amounts to finding the path of activations which minimizes the maximum margin violations given the path to the ground truth defined in step 3.
The CCCP algorithm has the desirable property of providing a monotonic decrease in the objective function at each iteration. In other words, the objective function value of problem (4) at W lt is greater than or equal to its value at W l t+1. Since layerwise optimization itself can be viewed as a block-coordinate algorithm for minimizing the learning objective (3), our overall algorithm provides guarantees of monotonic decrease until convergence. This is one of the main advantages of our approach compared to backpropagation and its variants, which fail to provide similar guarantees on the value of the objective function from one iteration to the next.
Algorithm 1 CCCP for parameter estimation of the l-th layer of the PL-CNN.
Require: Data set D = {(xi, yi), i ∈ [N ]}, fixed parameters {W ∪ W svm}\W l, initial estimate W l0.
1: t = 0 2: repeat 3: For each sample (xi, yi), find the best latent variable value by solving the following
problem: h∗i = argmax
h∈H (W lt ) >Ψ(xi, yi,h). (6)
4: Update the parameters by solving the following convex optimization problem:
W lt+1 = argmin W l
λ 2 ‖W l‖2F + 1 N N∑ i=1
max ȳi∈Y hi∈H
( ∆(ȳi, yi) + (W l)>Ψ(xi, ȳi,hi) ) −
( (W l)>Ψ(xi, yi,h ∗ i ) ) . (7)
5: t = t+1 6: until Objective function of problem (4) cannot be improved beyond a specified tolerance.
In order to solve the convex program (7), which corresponds to a structured SVM problem, we make use of the powerful BCFW algorithm (Lacoste-Julien et al., 2013) that solves its dual via conditional gradients. This has two main advantages: (i) as the dual is a smooth quadratic program, each iteration of BCFW provides a monotonic increase in its
objective; and (ii) the optimal step-size at each iteration can be computed analytically. This is once again in stark contrast to backpropagation, where the estimation of the step-size is still an active area of research (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015). As shown by Lacoste-Julien et al. (2013), given the current estimate of the parameters W l, the conditional gradient of the dual of program (7) with respect to a training sample (xi, yi) can be obtained by solving the following problem:
(ŷi, ĥi) = argmax ȳ∈Y,h∈H
(W l)>Ψ(xi, ȳ,h) + ∆(ȳ, yi). (8)
We refer the interested reader to (Lacoste-Julien et al., 2013) for further details.
The overall efficiency of the CCCP algorithm relies on our ability to solve problems (6) and (8). At first glance, these problems may appear to be computationally intractable as the latent space H can be very large, especially for layers close to the input (of the order of millions of dimensions for a typical network). However, the following proposition shows that both the problems can be solved efficiently using the forward and backward passes that are employed in backpropagation.
Proposition 3. Given the current estimate W l of the parameters for the l-th layer, as well as the parameter values of all the other fixed layers, problems (6) and (8) can be solved
using a forward pass on the network. Furthermore, the joint feature vectors Ψ(xi, ŷi, ĥi) and Ψ(xi, yi,h ∗ i ) can be computed using a backward pass on the network.
Sketch of the Proof. Recall that the latent space consists of the putative activations for each PL operation in the layers following the current one. Thus, intuitively, the maximization over the latent variables corresponds to finding the exact activations of all such PL operations. In other words, we need to identify the indices of the linear pieces that are used to compute the value of the PL function in the current state of the network. For a ReLU operation, this corresponds to estimating max{0, v}, where the input to the ReLU is a scalar v. Similarly, for a max-pool operation, this corresponds to estimating maxi{e>i u}, where u is the input vector to the max-pool. This is precisely the computation that the forward pass of backpropagation performs. Given the activations, the joint feature vector is the subgradient of the sample with respect to the current layer. Once again, this is precisely what is computed during the backward pass of the backpropagation algorithm.
An example is constructed in Appendix B to illustrate how to compute the feature vectors in practice.
4.3 Improving the BCFW Algorithm
As the BCFW algorithm was originally designed to solve a structured SVM problem, it requires further extensions to be suitable for training a PL-CNN. In what follows, we present three such extensions that improve the initialization, memory requirements and time complexity of the BCFW algorithm respectively.
Trust-Region for Initialization. The original BCFW algorithm starts with an initial parameter W l = 0 (that is, all the parameters are set to 0). The reason for this initialization is that it is possible to compute the dual variables that correspond to the 0 primal variable. However, since our algorithm visits each layer of the network several times, it would be desirable to initialize its parameters using its current value W tl . To this end, we introduce a trust-region in the constraints of problem (7), or equivalently, an `2 norm based proximal term in its objective function (Parikh & Boyd, 2014). The following proposition shows that this has the desired effect of initializing the BCFW algorithm close to the current parameter values.
Proposition 4. By adding a proximal term µ2 ‖W l −W lt‖2F to the objective function in (7), we can compute a feasible dual solution whose corresponding primal solution is equal to µ λ+µW l t . Furthermore, the addition of the proximal term still allows us to efficiently compute the conditional gradient using a forward-backward pass (proof in Appendix D).
In practice, we always choose a value of µ = 10λ: this yields an initialization of ' 0.9W lt which does not significantly change the value of the objective function.
Efficient Representation of Joint Feature Vectors. The BCFW algorithm requires us to store a linear combination of the feature vectors for each mini-batch. While this requirement is not too stringent for convolutional and multi-class SVM layers, where the dimensionality of the feature vectors is small, it becomes prohibitively expensive for dense layers. The following proposition prevents a blow-up in the memory requirements of BCFW.
Proposition 5. When optimizing dense layer l, if W l ∈ Rp×q, we can store a representation of the joint feature vectors Ψ(x, y,h) with vectors of size p in problems (6) and (7). This is in contrast to the näıve approach that requires them to be of size p× q.
Sketch of the Proof. By Proposition (3), the feature vectors are subgradients of the hinge loss function, which we loosely denote by η for this proof. Then by the chain rule: ∂η ∂W l
= ∂η ∂zl ∂zl ∂W l = ∂η ∂zl · ( zl−1 )T . Noting that zl−1 ∈ Rq is a forward pass up until layer l (independent of W l), we can store only ∂η ∂zl ∈ Rp and still reconstruct the full feature vector ∂η ∂W l by a forward pass and an outer product.
Reducing the Number of Constraints. In order to reduce the amount of time required for the BCFW algorithm to converge, we use the structure of H to simplify problem (7) to a much simpler problem. Specifically, since H represents the activations of the network for a given sample, it has a natural decomposition over the layers: H = H1 × ...×HL. We use this structure in the following observation.
Observation 1. Problem (7) can be approximately solved by optimizing the dual problem on increasingly large search spaces. In other words, we start with constraints of Y, followed by Y ×HL, then Y ×HL ×HL−1 and so on. The algorithm converges when the primal-dual gap is below tolerance.
The latent variables which are not optimized over are set to be the same as the ones selected for the ground truth. Experimentally, we observe that for convolutional layers (architectures in section 5), restricting the search space to Y yields a dual gap low enough to consider the problem has converged. This means that in practice for these layers, problem (7) can be solved by searching directions over the search space Y instead of the much larger Y × H. The intuition is that the norm of the difference-of-convex decomposition grows with the number of activations selected differently in the convex and concave parts (see Appendix A for the decomposition of piecewise linear functions). This compels the path of activations to be the same in the convex and the concave part to avoid large margin violations, especially for convolutional layers which are followed by numerous non-linearities at the max-pooling layers.
5 Experiments
Our experiments are designed to assess the ability of LW-SVM (Layer-Wise SVM, our method) and the SGD baselines to optimize problem (3). To compare LW-SVM with the state-of-the-art variants of backpropagation, we look at the training and testing accuracies as well as the training objective value. Unlike dropout, which effectively learns an ensemble model, we learn a single model using each baseline optimization algorithm. All experiments are conducted on a GPU (Nvidia Titan X) and use Theano (Bergstra et al., 2010; Bastien et al., 2012). We compare LW-SVM with Adagrad, Adadelta and Adam. For all data sets, we start at a good solution provided by these solvers and fine-tune it with LW-SVM. We then check whether a longer run of the SGD solver reaches the same level of performance.
The practical use of the LW-SVM algorithm needs choices at the three following levels: how to select the layer to optimize (i), when to stop the CCCP on each layer (ii) and when to stop the convex optimization at each inner iteration of the CCCP (iii). These choices are detailed in the next paragraph.
The layer-wise schedule of LW-SVM is as follows: as long as the validation accuracy increases, we perform passes from the end of the network (SVM) to the first layer (i). At each pass, each layer is optimized with one outer iteration of the CCCP (ii). The inner iterations are stopped when the dual objective function does not increase by more than 1% over an epoch (iii). We point out that the dual objective function is cheap to compute since we are maintaining its value at all time. By contrast, to compute the exact primal objective function requires a forward pass over the data set without any update.
5.1 MNIST Data Set
Data set & Architecture The training data set consists in 60,000 gray scale images of size 28× 28 with 10 classes, which we split into 50,000 samples for training and 10,000 for validating. The images are normalized, and we do not use any data augmentation. The architecture used for this experiment is shown in Figure 1.
Method The number of epochs is set to 200, 100 and 100 for Adagrad, Adadelta and Adam - Adagrad is given more epochs as we observed it took a longer time to converge. We then use LW-SVM and compare the results on training objective, training accuracy and testing accuracy. We also let the solvers run to up to 500 epochs to verify that we have not stopped the optimization prematurely. The regularization hyperparameter λ and the initial learning rate are chosen by cross-validation. λ is set to 0.001 for all solvers, and the initial learning rates can be found in Appendix C. For LW-SVM, λ is set to the same value as the baseline, and the proximal term µ to µ = 10λ = 0.01.
Results As Table 1 shows, LW-SVM systematically improves on all training objective, training accuracy and testing accuracy. In particular, it obtains the best testing accuracy when combined with Adadelta. Because each convex sub-problem is run up to sufficient convergence, the objective function of LW-SVM features of monotonic decrease at each iteration of the CCCP (blue curves in first row of Figure 2).
5.2 CIFAR Data Sets
Data sets & Architectures The CIFAR-10/100 data sets are comprised of 60,000 RGB natural images of size 32× 32 with 10/100 classes (Krizhevsky, 2009)). We split the training set into 45,000 training samples and 5,000 validation samples in both cases. The images are centered and normalized, and we do not use any data augmentation. To obtain a strong enough baseline, we employ (i) a pre-training with a softmax and cross-entropy loss and (ii) Batch-Normalization (BN) layers before each non-linearity.
We have experimentally found out that pre-training with a softmax layer followed by a cross-entropy loss led to better behavior and results than using an SVM loss alone. The baselines are trained with batch normalization. Once they have converged, the estimated mean and standard deviation are fixed like they would be at test time. Then batch normalization becomes a linear transformation, which can be handled by the LW-SVM algorithm. This allows us to compare LW-SVM with a baseline benefiting from batch normalization. Specifically, we use the architecture shown in Figure 3:
Method Again, the initial learning rates and regularization weight λ are obtained by cross-validation, and a value of 0.001 is obtained for λ for all solvers on both datasets. As before, µ is set to 10λ. The initial learning rates are reported in Appendix C. The layer schedule and convergence criteria are as described at the beginning of the section. For each SGD optimizer, we train the network for 10 epochs with a cross-entropy loss (preceded by a softmax layer). Then it is trained with an SVM loss (without softmax) for respectively 1000, 100 and 100 epochs for Adagrad, Adadelta and Adam. This amount is doubled to verify that the baselines are not harmed by a premature stopping. Results are presented in Tables 2 and 3.
Results It can be seen from this set of results that LW-SVM always improves over the solution of the SGD algorithm, for example on CIFAR-100, decreasing the objective value of Adam from 0.22 to 0.06, or improving the test accuracy of Adadelta from 84.4% to 86.6% on
CIFAR-10. The automatic step-size allows for a precise fine-tuning to optimize the training objective, while the regularization of the proximal term helps for better generalization.
5.3 ImageNet Data Set
Network Top-1 Accuracy Top-5 Accuracy VGG-16 (PT) 73.30% 91.33% VGG-16 (PT + LW-SVM) 73.81% 91.61%
Since the objective function penalizes the top-1 error, it is logical to observe that the improvement is most important on the top-1 accuracy. Importantly, having an efficient representation of feature vectors proves to be essential for such large networks: for instance, in the optimization of the first fully connected layer with a batch-size of 100, the use of our representation lowers the memory requirements of the BCFW algorithm from 7,600GB to 20GB, which can then fit in the memory of a powerful computer.
6 Discussion
We presented a novel layerwise optimization algorithm for a large and useful class of convolutional neural networks, which we term PL-CNNs. Our key observation is that the optimization of the parameters of one layer of a PL-CNN is equivalent to solving a latent structured SVM problem. As the problem is a DC program, it naturally lends itself to the iterative CCCP approach, which optimizes a convex structured SVM objective at each iteration. This allows us to leverage the advancements made in structured SVM optimization over the past decade to design a computationally feasible approach for learning PL-CNNs. Specifically, we use the BCFW algorithm and extend it to improve its initialization, memory requirements and time complexity. In particular, this allows our method to not require the tuning of any learning rate. Using the publicly available MNIST, CIFAR-10 and CIFAR-100 data sets, we show that our approach provides a boost for learning PL-CNNs over the state of the art backpropagation algorithms. Furthermore, we demonstrate scalability of the method with results on the ImageNet data set with a large network.
When the mean and standard deviation estimations of batch normalization are not fixed (unlike in our experiments with LW-SVM), batch normalization is not a piecewise linear transformation, and therefore cannot be used in conjunction with the BCFW algorithm for SVMs. However, it is difference-of-convex as it is a C2 function (Horst & Thoai, 1999). Incorporating a normalization scheme into our framework will be the object of future work. With our current methodology, LW-SVM algorithm can already be used on most standard architectures like VGG, Inception and ResNet-type architectures.
It is worth noting that other approaches for solving structured SVM problems, such as cutting-plane algorithms (Tsochantaridis et al., 2004; Joachims et al., 2009) and stochastic subgradient descent (Shalev-Shwartz et al., 2009), also rely on the efficiency of estimating the conditional gradient of the dual. Hence, all these methods are equally applicable to our setting. Indeed, the main strength of our approach is the establishment of a hitherto unknown connection between CNNs and latent structured SVMs. We believe that our observation will allow researchers to transfer the substantial existing knowledge of DC programs in general, and latent SVMs specifically, to produce the next generation of principled optimization algorithms for deep learning. In fact, there are already several such improvements that can be readily applied in our setting, which were not explored only due to a lack of time. This includes multi-plane variants of BCFW (Shah et al., 2015; Osokin et al., 2016), as well as generalizations of Frank-Wolfe such as partial linearization (Mohapatra et al., 2016).
Acknowledgments
This work was supported by the EPSRC AIMS CDT grant EP/L015987/1, the EPSRC Programme Grant Seebibyte EP/M013774/1 and Yougov. Many thanks to A. Desmaison, R. Bunel and D. Bouchacourt for the helpful discussions.
A Piecewise Linear Functions
Proof of Proposition (1) By the definition from (Melzer, 1986), we can write each function as the difference of two point-wise maxima of linear functions:
g(v) = max j∈[m+] {a>i v} − max k∈[m−] {b>j v} And ∀i ∈ [n], gi(u) = g+i (u)− g − i (u)
Where all the g+i , g − i are linear point-wise maxima of linear functions. Then:
f(u) = g([g1(u), · · · , gn(u)]>) = max j∈[m+] {a>j [g1(u), · · · , gn(u)]>} − max k∈[m−] {b>k [g1(u), · · · , gn(u)]>}
= max j∈[m+] { n∑ i=1 aj,igi(u) } − max k∈[m−] { n∑ i=1 bk,igi(u) }
= max j∈[m+] { n∑ i=1 aj,ig + i (u)− n∑ i=1 aj,ig − i (u) } − max k∈[m−] { n∑ i=1 bk,ig + i (u)− n∑ i=1 bk,ig − i (u) }
= max j∈[m+]  n∑ i=1 aj,ig + i (u) + ∑ j′∈[m+]\{j} n∑ i=1 aj,ig − i (u) − ∑ j′∈[m+] n∑ i=1 aj,ig − i (u)
− max k∈[m−]  n∑ i=1 bk,ig + i (u) + ∑ k′∈[m−]\{k} n∑ i=1 bk,ig − i (u) + ∑ k′∈[m−] n∑ i=1 bk,ig − i (u)
= max j∈[m+]  n∑ i=1 aj,ig + i (u) + ∑ j′∈[m+]\{j} n∑ i=1 aj,ig − i (u) + ∑ k′∈[m−] n∑ i=1 bk,ig − i (u)
−  max k∈[m−]  n∑ i=1 bk,ig + i (u) + ∑ k′∈[m−]\{k} n∑ i=1 bk,ig − i (u) + ∑ j′∈[m+] n∑ i=1 aj,ig − i (u)  = max j∈[m+]  n∑ i=1 aj,ig + i (u) + ∑ j′∈[m+]\{j} n∑ i=1 aj,ig − i (u) + ∑ k′∈[m−] n∑ i=1 bk,ig − i (u)
 − max k∈[m−]  n∑ i=1 bk,ig + i (u) + ∑ k′∈[m−]\{k} n∑ i=1 bk,ig − i (u) + ∑ j′∈[m+] n∑ i=1 aj,ig − i (u)
 In each line of the last equality, we recognize a pointwise maximum of a linear combination of pointwise maxima of linear functions. This constitutes a pointwise maximum of linear functions.
This derivation also extends equation (10) to the multi-dimensional case by showing an explicit DC decomposition of the output.
B Computing the Feature Vectors
We describe here how to compute the feature vectors in practice. To this end, we show how to construct two (intertwined) neural networks that decompose the objective function into a convex and a concave part. We call these Difference of Convex (DC) networks. Once the DC networks are defined, a standard forward and backward pass in the two networks yields the feature vectors for the convex and concave contribution to the objective function. First, we derive how to perform a DC decomposition in linear and non-linear layers, and then we construct an example of DC networks.
DC Decomposition in a Linear Layer Let W be the weights of a fixed linear layer. We introduce W+ = 12 (|W |+W ) and W − = 12 (|W | −W ). We can note that W + and W− have exclusively non-negative weights, and that W = W+ −W−. Say we have an input u with the DC decomposition (ucvx, uccv), that is: u = ucvx − uccv, where both ucvx and uccv are convex. Then we can decompose the output of the layer as:
W · u = (W+ · ucvx +W− · uccv)︸ ︷︷ ︸ convex − (W− · ucvx +W+ · uccv)︸ ︷︷ ︸ convex
(9)
DC Decomposition in a Piecewise Linear Activation Layer For simplicity purposes, we consider that the non-linear layer is a point-wise maximum across [K] scalar inputs, that is, for an input (uk)k∈[K] ∈ RK , the output is maxk∈[K] uk (the general multi-dimensional case can be found in Appendix A). We suppose that we have a DC decomposition (ucvxk , u ccv k ) for each input k. Then we can write the following decomposition for the output of the layer:
max k∈[K] uk = max k∈[K]
(ucvxk − uccvk )
= max k∈[K] ucvxk + ∑ i∈[K],i6=k uccvi  ︸ ︷︷ ︸
convex
− ∑ k∈[K]
uccvk︸ ︷︷ ︸ convex
(10)
In particular, for a ReLU, we can write:
max(ucvx − uccv, 0) = max(ucvx, uccv)︸ ︷︷ ︸ convex − uccv︸︷︷︸ convex
(11)
And for a Max-Pooling layer, one can easily verify that equation (10) is equivalent to:
MaxPool(ucvx − uccv) = MaxPool(ucvx − uccv) + SumPool(uccv)︸ ︷︷ ︸ convex −SumPool(uccv)︸ ︷︷ ︸ convex (12)
An Example of DC Networks We use the previous observations to obtain a DC decomposition in any layer. We now take the example of the neural network used for the experiments on the MNIST data set, and we show how to construct the two neural networks when optimizing W 1, the weights of the first convolutional layer. First let us recall the architecture without decomposition:
We want to optimize the first convolutional layer, therefore we fix all other parameters. Then we apply all operations as described in the previous paragraphs, which yields the DC networks in Figure 7.
The network graph in Figure 7 illustrates Proposition 3 for the optimization of W 1: suppose we are interested in f cvx(x,W 1), the convex part of the objective function for a given sample x, and we wish to obtain the feature vector needed to perform an update of BCFW. With a
forward pass, the oracle for the latent and label variables (ĥ, ŷ) is efficiently computed; and
with a backward pass, we obtain the corresponding feature vector Ψ(x, ŷ, ĥ). Indeed, we recall from problem (8) that (ĥ, ŷ) are the latent and label variables maximizing f cvx(x,W 1). Then given x, the forward pass in the DC networks sequentially solves the nested maximization: it maximizes the activation of the ReLU and MaxPooling units at each layer, thereby selecting
the best latent variable ĥ at each non-linear layer, and maximizes the output of the SVM
Concave Network Convex Network
Non-Decomposed Corresponding
Network
layer, thereby selecting the best label ŷ. At the end of the forward pass, f cvx(x,W 1) is
therefore available as the output of the convex network, and the feature vector Ψ(x, ŷ, ĥ) can be computed as a subgradient of f cvx(x,W 1) with respect to W 1.
Linearizing the concave part is equivalent to fixing the activations of the DC networks, which can be done by using a fixed copy of W 1 at the linearization point (all other weights being fixed anyway). Then one can re-use the above reasoning to obtain the feature vectors for the linearized concave part. Altogether, this methodology allows our algorithm to be implemented in any standard deep learning library (our implementation is available at http://github.com/oval-group/pl-cnn).
C Experimental Details
Hyper-parameters The hyper-parameters are obtained by cross-validation with a search on powers of 10. In this section, η will denote the initial learning rate. We denote the Softmax + Cross-Entropy loss by SCE, while SVM stands for the usual Support Vector Machines loss.
One may note that the hyper-parameters are the same for both CIFAR-10 and CIFAR-100 for each combination of solver and loss. This makes sense since the initial learning rate mainly depends on the architecture of the network (and not so much on which particular images are fed to this network), which is very similar for the experiments on the CIFAR-10 and CIFAR-100 data sets.
D SVM Formulation & Dual Derivation
Multi-Class SVM Suppose we are given a data set of N samples, for which every sample i has a feature vector φi ∈ Rd and a ground truth label yi ∈ Y. For every possible label ȳi ∈ Y, we introduce the augmented feature vector ψi(ȳi) ∈ R|Y|×d containing φi at index ȳi, −φi at index yi, and zeros everywhere else (then ψi(yi) is just a vector of zeros). We also define ∆(ȳi, yi) as the loss by choosing the output ȳi instead of the ground truth yi in our task. For classification, this is the zero-one loss for example.
The SVM optimization problem is formulated as:
min w,ξi
λ 2 ‖w‖2 + 1 N N∑ i=1 ξi
subject to: ∀i ∈ [N ], ∀ȳi ∈ Y, ξi ≥ wTψi(ȳi) + ∆(yi, ȳi)
Where λ is the regularization hyperparameter. We now add a proximal term to a given starting point w0:
min w,ξi
λ 2 ‖w‖2 + µ 2 ‖w − w0‖2 + 1 N N∑ i=1 ξi
subject to: ∀i ∈ [N ], ∀ȳi ∈ Y, ξi ≥ wTψi(ȳi) + ∆(yi, ȳi)
Factorizing the second-order polynomial in w, we obtain the equivalent problem (changed by a constant):
min w,ξi
λ+ µ 2 ‖w − µ λ+ µ w0‖2 + 1 N N∑ i=1 ξi
subject to: ∀i ∈ [N ], ∀ȳi ∈ Y, ξi ≥ wTψi(ȳi) + ∆(yi, ȳi)
For simplicity, we introduce the ratio ρ = µ
λ+ µ .
Dual Objective function The primal problem is:
min w,ξi
λ+ µ
2 ‖w − ρw0‖2 +
1
N N∑ i=1 ξi
subject to: ∀i ∈ [N ], ∀ȳi ∈ Y, ξi ≥ wTψi(ȳi) + ∆(yi, ȳi)
The dual problem can be written as:
max α≥0 min w,ξi
λ+ µ
2 ‖w − ρw0‖2 +
1
N N∑ i=1 ξi + 1 N N∑ i=1 ∑ ȳi∈Y αi(ȳi) ( ∆(yi, ȳi) + w Tψi(ȳi)− ξi )
Then we obtain the following KKT conditions:
∀i ∈ [N ], ∂· ∂ξi = 0 −→ ∑ ȳi∈Y αi(ȳi) = 1
∂· ∂w = 0 −→ w = ρw0 − 1 N 1 λ+ µ N∑ i=1 ∑ ȳi∈Y
αi(ȳi)ψi(ȳi)︸ ︷︷ ︸ Aα
We also introduce b = 1N (∆(yi, ȳi))i,ȳi . We define Pn(Y) as the sample-wise probability simplex:
u ∈ Pn(Y) if: ∀i ∈ [N ], ∀ȳi ∈ Y, ui(ȳi) ≥ 0 ∀i ∈ [N ], ∑ ȳi∈Y ui(ȳi) = 1
We inject back and simplify to:
max α∈Pn(Y) −(λ+ µ) 2 ‖Aα‖2 + µwT0 (Aα) + αT b
Finally:
min α∈Pn(Y) f(α)
Where:
f(α) , λ+ µ
2 ‖Aα‖2 − µwT0 (Aα)− αT b
BCFW derivation We write ∇(i)f the gradient of f w.r.t. the block (i) of variables in α, padded with zeros on blocks (j) for j 6= i. Similarly, A(i) and b(i) contain the rows of A and the elements of b for the block of coordinates (i) and zeros elsewhere. We can write:
∇(i)f(α) = (λ+ µ)AT(i)Aα− µA(i)w0 − b(i)
Then the search corner for the block of coordinates (i) is given by:
si = argmin s′i
( < s′i,∇(i)f(α) > ) = argmin
s′i
( (λ+ µ)αTATA(i)s ′ i − µwT0 A(i)s′i − bT(i)s ′ i ) We replace:
Aα = ρw0 − w
A(i)s ′ i =
1
N
1
λ+ µ ∑ ȳi∈Y s′i(ȳi)ψi(ȳi)
bT(i)s ′ i =
1
N ∑ ȳi∈Y s′i(ȳi)∆(ȳi, yi)
We then obtain:
si = argmin s′i −(w − ρw0)T ∑ ȳi∈Y s′i(ȳi)ψi(ȳi)− wT0 ρ ∑ ȳi∈Y s′i(ȳi)ψi(ȳi)− ∑ ȳi∈Y s′i(ȳi)∆(ȳi, yi)  = argmax
s′i wT ∑ ȳi∈Y s′i(ȳi)ψi(ȳi) + ∑ ȳi∈Y s′i(ȳi)∆(ȳi, yi)  As expected, this maximum is obtained by setting si to one at y ∗ i = argmax ȳi∈Y ( wTψi(ȳi) + ∆(ȳi, yi) ) and zeros elsewhere. We introduce the notation:
wi = −A(i)α(i) li = b T (i)α(i)
ws = −A(i)si ls = b T (i)si
Then we have:
ws = − 1
N
1
λ+ µ ψ(y∗i ) = −
1
N
1
λ+ µ
∂Hi(y ∗ i )
∂w
ls = 1
N ∆(yi, y
∗ i )
The optimal step size in the direction of the block of coordinates (i) is given by :
γ∗ = argmin γ f(α+ γ(si − αi))
The optimal step-size is given by:
γ∗ = < ∇(i)f(α), si − αi > (λ+ µ)‖A(si − αi)‖2
We introduce wd = −Aα = w − ρw0. Then we obtain:
γ∗ = (wi − ws)T (w − ρw0) + ρwT0 (wi − ws)− 1λ+µ (li − ls)
‖wi − ws‖2
= (wi − ws)Tw − 1λ+µ (li − ls)
‖wi − ws‖2
And the updates are the same as in standard BCFW:
Algorithm 2 BCFW with warm start
1: Let w(0) = w0, ∀i ∈ [N ], w(0)i = 0 2: Let l(0) = 0, ∀i ∈ [N ], l(0)i = 0 3: for k=0...K do 4: Pick i randomly in {1, .., n}
5: Get y∗i = argmax ȳi∈Y Hi(ȳi, w (k)) and ws = −
1
N
1
λ+ µ
∂Hi(y ∗ i , w (k))
∂w(k)
6: ls = 1 N∆(y ∗ i , yi) 7: γ = (wi − ws)Tw − 1λ+µ (li − ls)
‖wi − ws‖2 clipped to [0, 1]
8: w (k+1) i = (1− γ)w (k) i + γws 9: l (k+1) i = (1− γ)l (k) i + γls 10: w(k+1) = w(k) + w (k+1) i − w (k) i = w (k) + γ(w (k) s − w(k)i ) 11: l(k+1) = l(k) + l (k+1) i − l (k) i 12: end for
In particular, we have proved Proposition (4) in this section: w is initialized to ρw0 (KKT conditions), and the direction of the conditional gradient, ws, is given by ∂Hi(y ∗ i )
∂w , which is
independent of w0.
Note that the derivation of the Lagrangian dual has introduced a dual variable αi(ȳi) for each linear constraint of the SVM problem (this can be replaced by αi(hi, (ȳi)) if we consider latent variables). These dual variables indicate the complementary slackness not only for the output class ȳi, but also for each of the activation which defines a piece of the piecewise linear hinge loss. Therefore a choice of α defines a path of activations.
E Sensitivity of SGD Algorithms
Here we discuss some weaknesses of the SGD-based algorithms that we have encountered in practice for our learning objective function. These behaviors have been observed in the case of PL-CNNs, and generally may not appear in different architectures (in particular the failure to learn with high regularization goes away with the use of batch normalization layers).
E.1 Initial Learning Rate
As mentioned in the experiments section, the choice of the initial learning rate is critical for good performance of all Adagrad, Adadelta and Adam. When the learning rate is too high, the network does not learn anything and the training and validating accuracies are stuck at random level. When it is too low, the network may take a considerably greater number of epochs to converge.
E.2 Failures to Learn
Regularization When the regularization hyper-parameter λ is set to a value of 0.01 or higher on CIFAR-10, SGD solvers get trapped in a local minimum and fail to learn. The SGD solvers indeed fall in the local minimum of shutting down all activations on ReLUs, which provide zero-valued feature vector to the SVM loss layer (and a hinge loss of one). As a consequence, no information can be back-propagated. We plot this behavior below:
In this situation, the network is at a bad saddle point (note that the training and validation accuracies are stuck at random levels). Our algorithm does not fall into such bad situations, however it is not able to get out of it either: each layer is at a pathological critical point of its own objective function, which makes our algorithm unable to escape from it.
With a lower initial learning rate, the evolution is slower, but eventually the solver goes back to the bad situation presented above.
Biases The same failing behavior as above has been observed when not using the biases in the network. Again our algorithm is robust to this change.

This paper proposes a new approaches for optimizing the objective of CNNs. The proposed method uses a lay-wise optimization, i.e. at each step, it optimizes the parameters in one layer of CNN while fixing the parameters in other layers. The key insight of this paper is that, for a large class of CNNs, the optimization problem at a particular can be formulated as optimizing a piecewise linear (PL) function. This PL function optimization happens to be the optimization problem commonly encountered in latent structural SVM. This connection allows this paper to borrows ideas from the latent structural SVM literature, in particular concave-convex procedure, to learn the parameters of CNNs.
Overall, the paper is well-written. Traditional, CNNs and structural SVMs are almost two separate research communties. The connection of CNNs to latent structural SVM is interesting, and might bridge the gap and facilitate the transferring of ideas between these two camps.
Of course, the proposed method also has some limitations. 1) It is limited to layer-wise optimization. Nowadays layer-wise optimization is essentially a coordinate descent algorithm and is not really a competitive strategy in learning CNNs. When you choose layer-wise optimization, you already lose something in terms of optimizing the objective (since you are using coordinate descent, instead of gradient descent). Of course, you also gain something since now you can guarantee that each coordinate descent step always improve the objective. It is not clear to me how the loss/gain balances each other. 2) This paper focues on improving the optimization of CNN objective. However, we all know that a better objective does not necessarily correspond to a good model (e.g. due to overfitting). Although the SGD with backprop in standard CNN learning does not always improve the solution of the objective (unlike the proposed method in this paper), but to me, this might be a good thing since it can prevent overfitting (the goal of learning is not to get better solution for the optimization problem in the first place -- the optimization problem is merely a proxy to learn a model with good generalization ability).
The experiment is a bit weak.
1) Only CIFAR10 is used. This is a very small dataset by today's standard, while CNNs are typically used in large-scale datasets, such as ImageNet. It is not clear whether the conclusions of this paper still hold when applied on ImageNet.
2) This paper only compares with a crippled variant of SGD (without batch normalization, dropout, etc). Although this paper mentions that the reason is that it wants to focus on optimization. But I mentioned earlier, SGD is not designed to purely obtain the best solution that optimizes the objective, the goal of SGD is to reasonably optimize the objective, while preventing overfitting. So the comparison to SGD purely in terms of the optimization is that meaningful in the first place.


Adversarial examples are malicious inputs designed to fool machine learning models. They often transfer from one model to another, allowing attackers to mount black box attacks without knowledge of the target model’s parameters. Adversarial training is the process of explicitly training a model on adversarial examples, in order to make it more robust to attack or to reduce its test error on clean inputs. So far, adversarial training has primarily been applied to small problems. In this research, we apply adversarial training to ImageNet (Russakovsky et al., 2014). Our contributions include: (1) recommendations for how to succesfully scale adversarial training to large models and datasets, (2) the observation that adversarial training confers robustness to single-step attack methods, (3) the finding that multi-step attack methods are somewhat less transferable than singlestep attack methods, so single-step attacks are the best for mounting black-box attacks, and (4) resolution of a “label leaking” effect that causes adversarially trained models to perform better on adversarial examples than on clean examples, because the adversarial example construction process uses the true label and the model can learn to exploit regularities in the construction process.
1 INTRODUCTION
It has been shown that machine learning models are often vulnerable to adversarial manipulation of their input intended to cause incorrect classification (Dalvi et al., 2004). In particular, neural networks and many other categories of machine learning models are highly vulnerable to attacks based on small modifications of the input to the model at test time (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2014; Papernot et al., 2016b).
The problem can be summarized as follows. Let’s say there is a machine learning system M and input sample C which we call a clean example. Let’s assume that sample C is correctly classified by the machine learning system, i.e. M(C) = ytrue. It’s possible to construct an adversarial example A which is perceptually indistinguishable from C but is classified incorrectly, i.e. M(A) 6= ytrue. These adversarial examples are misclassified far more often than examples that have been perturbed by noise, even if the magnitude of the noise is much larger than the magnitude of the adversarial perturbation (Szegedy et al., 2014).
Adversarial examples pose potential security threats for practical machine learning applications. In particular, Szegedy et al. (2014) showed that an adversarial example that was designed to be misclassified by a model M1 is often also misclassified by a model M2. This adversarial example transferability property means that it is possible to generate adversarial examples and perform a misclassification attack on a machine learning system without access to the underlying model. Papernot et al. (2016a) and Papernot et al. (2016b) demonstrated such attacks in realistic scenarios.
It has been shown (Goodfellow et al., 2014; Huang et al., 2015) that injecting adversarial examples into the training set (also called adversarial training) could increase robustness of neural networks to adversarial examples. Another existing approach is to use defensive distillation to train the network (Papernot et al., 2015). However all prior work studies defense measures only on relatively small datasets like MNIST and CIFAR10. Some concurrent work studies attack mechanisms on ImageNet (Rozsa et al., 2016), focusing on the question of how well adversarial examples transfer between different types of models, while we focus on defenses and studying how well different types of adversarial example generation procedures transfer between relatively similar models.
In this paper we studied adversarial training of Inception models trained on ImageNet. The contributions of this paper are the following:
• We successfully used adversarial training to train an Inception v3 model (Szegedy et al., 2015) on ImageNet dataset (Russakovsky et al., 2014) and to significantly increase robustness against adversarial examples generated by the fast gradient sign method (Goodfellow et al., 2014) as well as other one-step methods.
• We demonstrated that different types of adversarial examples tend to have different transferability properties between models. In particular we observed that those adversarial examples which are harder to resist using adversarial training are less likely to be transferrable between models.
• We showed that models which have higher capacity (i.e. number of parameters) tend to be more robust to adversarial examples compared to lower capacity model of the same architecture. This provides additional cue which could help building more robust models.
• We also observed an interesting property we call “label leaking”. Adversarial examples constructed with a single-step method making use of the true labels may be easier to classify than clean adversarial examples, because an adversarially trained model can learn to exploit regularities in the adversarial example construction process. This suggests using adversarial example construction processes that do not make use of the true label.
The rest of the paper is structured as follows: In section 2 we review different methods to generate adversarial examples. Section 3 describes details of our adversarial training algorithm. Finally, section 4 describes our experiments and results of adversarial training.
2 METHODS GENERATING ADVERSARIAL EXAMPLES
2.1 TERMINOLOGY AND NOTATION
In this paper we use the following notation and terminology regarding adversarial examples:
1. X , the clean image — unmodified image from the dataset (either train or test set).
2. Xadv , the adversarial image: the output of any procedure intended to produce an approximate worst-case modification of the clean image. We sometimes call this a candidate adversarial image to emphasize that an adversarial image is not necessarily misclassified by the neural network.
3. Misclassified adversarial image — candidate adversarial image which is misclassified by the neural network. In addition we are typically interested only in those misclassified adversarial images when the corresponding clean image is correctly classified.
4. : The size of the adversarial perturbation. In most cases, we require the L∞ norm of the perturbation to be less than , as done by Goodfellow et al. (2014). We always specify in terms of pixel values in the range [0, 255]. Note that some other work on adversarial examples minimizes the size of the perturbation rather than imposing a constraint on the size of the perturbation (Szegedy et al., 2014).
5. The cost function used to train the model is denoted J(X, ytrue).
6. ClipX, (A) denotes element-wise clipping A, with Ai,j clipped to the range [Xi,j − ,Xi,j + ].
7. One-step methods of adversarial example generation generate a candidate adversarial image after computing only one gradient. They are often based on finding the optimal perturbation of a linear approximation of the cost or model. Iterative methods apply many gradient updates. They typically do not rely on any approximation of the model and typically produce more harmful adversarial examples when run for more iterations.
2.2 ATTACK METHODS
We study a variety of attack methods:
Fast gradient sign method Goodfellow et al. (2014) proposed the fast gradient sign method (FGSM) as a simple way to generate adversarial examples:
Xadv = X + sign ( ∇XJ(X, ytrue) ) (1)
This method is simple and computationally efficient compared to more complex methods like LBFGS (Szegedy et al., 2014), however it usually has a lower success rate. On ImageNet, top-1 error rate on candidate adversarial images for the FGSM is about 63%− 69% for ∈ [2, 32].
One-step target class methods FGSM finds adversarial perturbations which increase the value of the loss function. An alternative approach is to maximize probability p(ytarget | X) of some specific target class ytarget which is unlikely to be the true class for a given image. For a neural network with cross-entropy loss this will lead to the following formula for the one-step target class method:
Xadv = X − sign ( ∇XJ(X, ytarget) ) (2)
As a target class we can use the least likely class predicted by the network yLL = argminy { p(y |
X) }
, as suggested by Kurakin et al. (2016). In such case we refer to this method as one-step least likely class or just “step l.l.” Alternatively we can use a random class as target class. In such a case we refer to this method as “step rnd.”.
Basic iterative method A straightforward extension of FGSM is to apply it multiple times with small step size:
Xadv0 = X, X adv N+1 = ClipX, { XadvN + α sign ( ∇XJ(XadvN , ytrue) )} In our experiments we used α = 1, i.e. we changed the value of each pixel only by 1 on each step. We selected the number of iterations to be min( + 4, 1.25 ). See more information on this method in Kurakin et al. (2016). Below we refer to this method as “iter. basic” method.
Iterative least-likely class method By running multiple iterations of the “step l.l.” method we can get adversarial examples which are misclassified in more than 99% of the cases:
Xadv0 = X, X adv N+1 = ClipX, { XadvN − α sign ( ∇XJ(XadvN , yLL) )} α and number of iterations were selected in the same way as for the basic iterative method. Below we refer to this method as the “iter. l.l.”.
3 ADVERSARIAL TRAINING
The basic idea of adversarial training is to inject adversarial examples into the training set, continually generating new adversarial examples at every step of training (Goodfellow et al., 2014). Adversarial training was originally developed for small models that did not use batch normalization. To scale adversarial training to ImageNet, we recommend using batch normalization (Ioffe & Szegedy, 2015). To do so successfully, we found that it was important for examples to be grouped into batches containing both normal and adversarial examples before taking each training step, as described in algorithm 1.
We use a loss function that allows independent control of the number and relative weight of adversarial examples in each batch:
Loss = 1
(m− k) + λk ( ∑ i∈CLEAN L(Xi|yi) + λ ∑ i∈ADV L(Xadvi |yi) ) where L(X|y) is a loss on a single example X with true class y; m is total number of training examples in the minibatch; k is number of adversarial examples in the minibatch and λ is a parameter which controls the relative weight of adversarial examples in the loss. We used λ = 0.3, m = 32,
Algorithm 1 Adversarial training of network N . Size of the training minibatch is m. Number of adversarial images in the minibatch is k.
1: Randomly initialize network N 2: repeat 3: Read minibatch B = {X1, . . . , Xm} from training set 4: Generate k adversarial examples {X1adv, . . . , Xkadv} from corresponding clean examples {X1, . . . , Xk} using current state of the network N 5: Make new minibatch B′ = {X1adv, . . . , Xkadv, Xk+1, . . . , Xm} 6: Do one training step of network N using minibatch B′ 7: until training converged
and k = 16. Note that we replace each clean example with its adversarial counterpart, for a total minibatch size of 32, which is a departure from previous approaches to adversarial training.
Fraction and weight of adversarial examples which we used in each minibatch differs from Huang et al. (2015) where authors replaced entire minibatch with adversarial examples. However their experiments was done on smaller datasets (MNIST and CIFAR-10) in which case adversarial training does not lead to decrease of accuracy on clean images. We found that our approach works better for ImageNet models (corresponding comparative experiments could be found in Appendix E).
We observed that if we fix during training then networks become robust only to that specific value of . We therefore recommend choosing randomly, independently for each training example. In our experiments we achieved best results when magnitudes were drawn from a truncated normal distribution defined in interval [0, 16] with underlying normal distribution N(µ = 0, σ = 8).1
4 EXPERIMENTS
We adversarially trained an Inception v3 model (Szegedy et al., 2015) on ImageNet. All experiments were done using synchronous distributed training on 50 machines, with a minibatch of 32 examples on each machine. We observed that the network tends to reach maximum accuracy at around 130k− 150k iterations. If we continue training beyond 150k iterations then eventually accuracy might decrease by a fraction of a percent. Thus we ran experiments for around 150k iterations and then used the obtained accuracy as the final result of the experiment.
Similar to Szegedy et al. (2015) we used RMSProp optimizer for training. We used a learning rate of 0.045 except where otherwise indicated.
We looked at interaction of adversarial training and other forms or regularization (dropout, label smoothing and weight decay). By default training of Inception v3 model uses all three of them. We noticed that disabling label smoothing and/or dropout leads to small decrease of accuracy on clean examples (by 0.1% - 0.5% for top 1 accuracy) and small increase of accuracy on adversarial examples (by 1% - 1.5% for top 1 accuracy). On the other hand reducing weight decay leads to decrease of accuracy on both clean and adversarial examples.
We experimented with delaying adversarial training by 0, 10k, 20k and 40k iterations. In such case we used only clean examples during the first N training iterations and after N iterations included both clean and adversarial examples in the minibatch. We noticed that delaying adversarial training has almost no effect on accuracy on clean examples (difference in accuracy within 0.2%) after sufficient number of training iterations (more than 70k in our case). At the same time we noticed that larger delays of adversarial training might cause up to 4% decline of accuracy on adversarial examples with high magnitude of adversarial perturbations. For small 10k delay changes of accuracy was not statistically significant to recommend against it. We used a delay of 10k because this allowed us to reuse the same partially trained model as a starting point for many different experiments.
For evaluation we used the ImageNet validation set which contains 50, 000 images and does not intersect with the training set.
1 In TensorFlow this could be achieved by tf.abs(tf.truncated normal(shape, mean=0, stddev=8)).
4.1 RESULTS OF ADVERSARIAL TRAINING
We experimented with adversarial training using several types of one-step methods. We found that adversarial training using any type of one-step method increases robustness to all types of one-step adversarial examples that we tested. However there is still a gap between accuracy on clean and adversarial examples which could vary depending on the combination of methods used for training and evaluation.
Adversarial training caused a slight (less than 1%) decrease of accuracy on clean examples in our ImageNet experiments. This differs from results of adversarial training reported previously, where adversarial training increased accuracy on the test set (Goodfellow et al., 2014; Miyato et al., 2016b;a). One possible explanation is that adversarial training acts as a regularizer. For datasets with few labeled examples where overfitting is the primary concern, adversarial training reduces test error. For datasets like ImageNet where state-of-the-art models typically have high training set error, adding a regularizer like adversarial training can increase training set error more than it decreases the gap between training and test set error. Our results suggest that adversarial training should be employed in two scenarios:
1. When a model is overfitting, and a regularizer is required. 2. When security against adversarial examples is a concern. In this case, adversarial training
is the method that provides the most security of any known defense, while losing only a small amount of accuracy.
By comparing different one-step methods for adversarial training we observed that the best results in terms or accuracy on test set are achieved using “step l.l.” or “step rnd.” method. Moreover using these two methods helped the model to become robust to adversarial examples generated by other one-step methods. Thus for final experiments we used “step l.l.” adversarial method.
For brevity we omitted a detailed comparison of different one-step methods here, but the reader can find it in Appendix A.
Results of adversarial training using “step l.l.” method are provided in Table 1. As it can be seen from the table we were able to significantly increase top-1 and top-5 accuracy on adversarial examples (up to 74% and 92% correspondingly) to make it to be on par with accuracy on clean images. However we lost about 0.8% accuracy on clean examples.
We were able to slightly reduce the gap in the accuracy on clean images by slightly increasing the size of the model. This was done by adding two additional Inception blocks to the model. For specific details about Inception blocks refer to Szegedy et al. (2015).
Unfortunately, training on one-step adversarial examples does not confer robustness to iterative adversarial examples, as shown in Table 2.
We also tried to use iterative adversarial examples during training, however we were unable to gain any benefits out of it. It is computationally costly and we were not able to obtain robustness to adversarial examples or to prevent the procedure from reducing the accuracy on clean examples significantly. It is possible that much larger models are necessary to achieve robustness to such a large class of inputs.
4.2 LABEL LEAKING
We discovered a label leaking effect: when a model is trained on FGSM adversarial examples and then evaluated using FGSM adversarial examples, the accuracy on adversarial images becomes much higher than the accuracy on clean images (see Table 3). This effect also occurs (but to a lesser degree) when using other one-step methods that require the true label as input.
We say that label for specific example has been leaked if and only if the model classifies an adversarial example correctly when that adversarial example is generated using the true label but misclassifies a corresponding adversarial example that was created without using the true label. If too many labels has been leaked then accuracy on adversarial examples might become bigger than accuracy on clean examples which we observed on ImageNet dataset.
We believe that the effect occurs because one-step methods that use the true label perform a very simple and predictable transformation that the model can learn to recognize. The adversarial example construction process thus inadvertently leaks information about the true label into the input. We found that the effect vanishes if we use adversarial example construction processes that do not use the true label. The effect also vanishes if an iterative method is used, presumably because the output of an iterative process is more diverse and less predictable than the output of a one-step process.
Overall due to the label leaking effect, we do not recommend to use FGSM or other methods defined with respect to the true class label to evaluate robustness to adversarial examples; we recommend to use other one-step methods that do not directly access the label instead.
We recommend to replace the true label with the most likely label predicted by the model. Alternately, one can maximize the cross-entropy between the full distribution over all predicted labels given the clean input and the distribution over all predicted labels given the perturbed input (Miyato et al., 2016b).
We revisited the adversarially trained MNIST classifier from Goodfellow et al. (2014) and found that it too leaks labels. The most labels are leaked with = 0.3 on MNIST data in [0, 1]. With that , the model leaks 79 labels on the test set of 10,000 examples. However, the amount of label leaking is small compared to the amount of error caused by adversarial examples. The error rate on adversarial examples exceeds the error rate on clean examples for ∈ {.05, .1, .25, .3, .4, .45, .5}. This explains why the label leaking effect was not noticed earlier.
4.3 INFLUENCE OF MODEL CAPACITY ON ADVERSARIAL ROBUSTNESS
We studied how the size of the model (in terms of number of parameters) could affect robustness to adversarial examples. We picked Inception v3 as a base model and varied its size by changing the number of filters in each convolution.
For each experiment we picked a scale factor ρ and multiplied the number of filters in each convolution by ρ. In other words ρ = 1 means unchanged Inception v3, ρ = 0.5 means Inception with half of the usual number of filters in convolutions, etc . . . For each chosen ρ we trained two independent models: one with adversarial training and another without. Then we evaluated accuracy on clean and adversarial examples for both trained models. We have run these experiments for ρ ∈ [0.5, 2.0]. In earlier experiments (Table 1) we found that deeper models benefit more from adversarial training. The increased depth changed many aspects of the model architecture. These experiments varying ρ examine the effect in a more controlled setting, where the architecture remains constant except for the number of feature maps in each layer.
In all experiments we observed that accuracy on clean images kept increasing with increase of ρ, though its increase slowed down as ρ became bigger. Thus as a measure of robustness we used the ratio of accuracy on adversarial images to accuracy on clean images because an increase of this ratio means that the gap between accuracy on adversarial and clean images becomes smaller. If this ratio reaches 1 then the accuracy on adversarial images is the same as on clean ones. For a successful adversarial example construction technique, we would never expect this ratio to exceed 1, since this would imply that the adversary is actually helpful. Some defective adversarial example construction techniques, such as those suffering from label leaking, can inadvertently produce a ratio greater than 1.
Results with ratios of accuracy for various adversarial methods and are provided in Fig. 1.
For models without adversarial training, we observed that there is an optimal value of ρ yielding best robustness. Models that are too large or too small perform worse. This may indicate that models become more robust to adversarial examples until they become large enough to overfit in some respect.
For adversarially trained models, we found that robustness consistently increases with increases in model size. We were not able to train large enough models to find when this process ends, but we did find that models with twice the normal size have an accuracy ratio approaching 1 for one-step adversarial examples. When evaluated on iterative adversarial examples, the trend toward increasing robustness with increasing size remains but has some exceptions. Also, none of our models was large enough to approach an accuracy ratio of 1 in this regime.
Overall we recommend exploring increase of accuracy (along with adversarial training) as a measure to improve robustness to adversarial examples.
4.4 TRANSFERABILITY OF ADVERSARIAL EXAMPLES
From a security perspective, an important property of adversarial examples is that they tend to transfer from one model to another, enabling an attacker in the black-box scenario to create adversarial
examples for their own substitute model, then deploy those adversarial examples to fool a target model (Szegedy et al., 2014; Goodfellow et al., 2014; Papernot et al., 2016b).
We studied transferability of adversarial examples between the following models: two copies of normal Inception v3 (with different random initializations and order or training examples), Inception v4 (Szegedy et al., 2016) and Inception v3 which uses ELU activation (Clevert et al., 2015) instead of Relu2. All of these models were independently trained from scratch until they achieved maximum accuracy.
2 We achieved 78.0% top 1 and 94.1% top 5 accuracy on Inception v3 with ELU activations, which is comparable with accuracy of Inception v3 model with Relu activations.
In each experiment we fixed the source and target networks, constructed adversarial examples from 1000 randomly sampled clean images from the test set using the source network and performed classification of all of them using both source and target networks. These experiments were done independently for different adversarial methods.
We measured transferability using the following criteria. Among 1000 images we picked only misclassified adversarial example for the source model (i.e. clean classified correctly, adversarial misclassified) and measured what fraction of them were misclassified by the target model.
Transferability results for all combinations of models and = 16 are provided in Table 4. Results for various but fixed source and target model are provided in Fig. 2.
As can be seen from the results, FGSM adversarial examples are the most transferable, while “iter l.l.” are the least. On the other hand “iter l.l.” method is able to fool the network in more than 99% cases (top 1 accuracy), while FGSM is the least likely to fool the network. This suggests that there might be an inverse relationship between transferability of specific method and ability of the method to fool the network. We haven’t studied this phenomenon further, but one possible explanation could be the fact that iterative methods tend to overfit to specific network parameters.
In addition, we observed that for each of the considered methods transfer rate is increasing with increase of (see Fig. 2). Thus potential adversary performing a black-box attack have an incentive to use higher to increase the chance of success of the attack.
5 CONCLUSION
In this paper we studied how to increase robustness to adversarial examples of large models (Inception v3) trained on large dataset (ImageNet). We showed that adversarial training provides robustness to adversarial examples generated using one-step methods. While adversarial training didn’t help much against iterative methods we observed that adversarial examples generated by iterative methods are less likely to be transferred between networks, which provides indirect robustness against black box adversarial attacks. In addition we observed that increase of model capacity could also help to increase robustness to adversarial examples especially when used in conjunction with adversarial training. Finally we discovered the effect of label leaking which resulted in higher accuracy on FGSM adversarial examples compared to clean examples when the network was adversarially trained.
A COMPARISON OF ONE-STEP ADVERSARIAL METHODS
In addition to FGSM and “step l.l.” methods we explored several other one-step adversarial methods both for training and evaluation. Generally all of these methods can be separated into two large categories. Methods which try to maximize the loss (similar to FGSM) are in the first category. The second category contains methods which try to maximize the probability of a specific target class (similar to “step l.l.”). We also tried to use different types of random noise instead of adversarial images, but random noise didn’t help with robustness against adversarial examples.
The full list of one-step methods we tried is as follows:
• Methods increasing loss function J – FGSM (described in details in Section 2.2):
Xadv = X + sign ( ∇XJ(X, ytrue) ) – FGSM-pred or fast method with predicted class. It is similar to FGSM but uses the
label of the class predicted by the network instead of true class ytrue. – “Fast entropy” or fast method designed to maximize the entropy of the predicted dis-
tribution, thereby causing the model to become less certain of the predicted class. – “Fast grad. L2” is similar to FGSM but uses the value of gradient instead of its sign.
The value of gradient is normalized to have unit L2 norm:
Xadv = X + ∇XJ(X, ytrue)∥∥∇XJ(X, ytrue)∥∥2
Miyato et al. (2016b) advocate this method. – “Fast grad. L∞” is similar to “fast grad. L2” but uses L∞ norm for normalization.
• Methods increasing the probability of the selected target class – “Step l.l.” is one-step towards least likely class (also described in Section 2.2):
Xadv = X − sign ( ∇XJ(X, ytarget) ) where ytarget = argminy { p(y |X) } is least likely class prediction by the network. – “Step rnd.” is similar to “step l.l.” but uses random class instead of least likely class. • Random perturbations
– Sign of random perturbation. This is an attempt to construct random perturbation which has similar structure to perturbations generated by FGSM:
Xadv = X + sign ( N )
where N is random normal variable with zero mean and identity covariance matrix. – Random truncated normal perturbation with zero mean and 0.5 standard deviation
defined on [− , ] and uncorrelated pixels, which leads to the following formula for perturbed images: Xadv = X + T where T is a random variable with truncated normal distribution.
Overall, we observed that using only one of these single step methods during adversarial training is sufficient to gain robustness to all of them. Fig. 3 shows accuracy on various one-step adversarial examples when the network was trained using only “step l.l.” method.
At the same time we observed that not all one-step methods are equally good for adversarial training, as shown in Table 5. The best results (achieving both good accuracy on clean data and good accuracy on adversarial inputs) were obtained when adversarial training was done using “step l.l.” or “step rnd.” methods.
B ADDITIONAL RESULTS WITH SIZE OF THE MODEL
Section 4.3 contains details regarding the influence of size of the model on robustness to adversarial examples. Here we provide additional Figure 4 which shows robustness calculated using top 5 accuracy. Generally it exhibits the same properties as the corresponding plots for top 1 accuracy.
C ADDITIONAL RESULTS ON TRANSFERABILITY
Section 4.4 contains results with transfer rate of various adversarial examples between models. In addition to transfer rate computed only on misclassified adversarial examples it is also interesting to observe the error rate of all candidate adversarial examples generated for one model and classified by other model.
This result might be interesting because it models the following attack. Instead of trying to pick “good” adversarial images an adversary tries to modify all available images in order to get as much misclassified images as possible.
To compute the error rate we randomly generated 1000 adversarial images using the source model and then classified them using the target model. Results for various models, adversarial methods
and fixed = 16 are provided in Table 6. Results for fixed source and target models and various are provided in Fig. 5.
Overall the error rate of transferred adversarial examples exhibits the same behavior as the transfer rate described in Section 4.4.
D RESULTS WITH DIFFERENT ACTIVATION FUNCTIONS
We evaluated robustness to adversarial examples when the network was trained using various nonlinear activation functions instead of the standard relu activation when used with adversarial training on “step l.l.” adversarial images. We tried to use following activation functions instead of relu:
• tanh(x)
• relu6(x) = min(relu(x), 6)
• ReluDecayβ(x) = relu(x)1+βrelu(x)2 for β ∈ {0.1, 0.01, 0.001}
Training converged using all of these activations, however test performance was not necessarily the same as with relu.
tanh and ReluDecayβ=0.1 lose about 2%-3% of accuracy on clean examples and about 10%-20% on “step l.l.” adversarial examples. relu6,ReluDecayβ=0.01 andReluDecayβ=0.001 demonstrated similar accuracy (within ±1%) to relu on clean images and few percent loss of accuracy on “step l.l.” images. At the same time all non-linear activation functions increased classification accuracy on some of the iterative adversarial images. Detailed results are provided in Table 7.
Overall non linear activation functions could be used as an additional measure of defense against iterative adversarial images.
E RESULTS WITH DIFFERENT NUMBER OF ADVERSARIAL EXAMPLES IN THE MINIBATCH
We studied how number of adversarial examples k in the minibatch affect accuracy on clean and adversarial examples. Results are summarized in Table 8.
Overall we noticed that increase of k lead to increase of accuracy on adversarial examples and to decrease of accuracy on clean examples. At the same having more than half of adversarial examples in the minibatch (which correspond to k > 16 in our case) does not provide significant improvement of accuracy on adversarial images, however lead to up to 1% of additional decrease of accuracy
on clean images. Thus for most experiments in the paper we have chosen k = 16 as a reasonable trade-off between accuracy on clean and adversarial images.

This paper investigate the phenomenon of the adversarial examples and the adversarial training on the dataset of ImageNet. While the final conclusions are still vague, this paper raises several noteworthy finding from its experiments.
The paper is well written and easy to follow. Although I still have some concerns about the paper (see the comments below), this paper has good contributions and worth to publish.
Pros:
For the first time in the literature, this paper proposed the concept of ‘label leaking’. Although its effect only becomes significant when the dataset is large, it should be carefully handled in the future research works along this line.
Using the ratio of 'clean accuracy' over ‘adversarial accuracy’ as the measure of robust is more reasonable compared to the existing works in the literature.
Cons:
Although the conclusions of the paper are based on the experiments on ImageNet, the title of the paper seems a little misleading. I consider Section 4 as the main contribution of the paper. Note that Section 4.3 and Section 4.4 are not specific to large-scale dataset, thus emphasizing the ‘large-scale’ in the title and in the introduction seems improper.
Basically all the conclusions of the paper are made based on observing the experimental results. Further tests should have been performed to verify these hypotheses. Without that, the conclusions of the paper seems rushy. For example, one dataset of imageNet can not infer the conclusions for all large-scale datasets.