ADEF: AN ITERATIVE ALGORITHM TO CONSTRUCT ADVERSARIAL DEFORMATIONS
While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood. In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image. In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step. We demonstrate our results on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101.
The paper proposes a new way to construct adversarial examples: do not change the intensity of the input image directly, but deform the image plane (i.e. compose the image with Id + tau where tau is a small amplitude vector field).
The paper originates from a document provably written in late 2017, which is before the deposit on arXiv of another article (by different authors, early 2018) which was later accepted to ICLR 2018 [Xiao and al.]. This remark is important in that it changes my rating of the paper (being more indulgent with papers proposing new ideas, as otherwise the novelty is rather low compared to [Xiao and al.]).
Pros:
- the paper is well written, very easy to read, well explained (and better formalized than [Xiao and al.]);
- the idea of deforming images is new (if we forget about [Xiao and al.]) and simple;
- experiments show what such a technique can achieve on MNIST and ImageNet. Interestingly, one can see on MNIST the parts of the numbers that the adversarial attack is trying to delete/create.
Cons:
- the paper is a bit weak, in that it is not very dense, and in that there is not much more content than the initial idea;
- for instance, more discussions about the results obtained could have been appreciated (such as my remark above about MNIST);
- for instance, a study of the impact of the regularization would have been interesting (how does the sigma of the Gaussian smoothing affect the type of adversarial attacks obtained and their performance -- is it possible to fool the network with [very] smooth deformations?);
- for instance, what about generating adversarial examples for which the network would be fully (wrongly) confident? (instead of just borderline unsure); etc.
- The interpolation scheme (how is defined the intensity I(x,y) for a non-integer location (x,y) within the image I) is rather important (linear interpolation, etc.) and should be at least mentioned in the main paper, and at best studied (it might impact the gradient descent path and the results);
- question: does the algorithm converge? could there be a proof of this? This is not obvious, as the objective potentially changes with time (selection of the current m best indices k of |F_k - F_l|). Also, the final overshoot factor (1+eta) is not very elegant, and not guaranteed to perform well if tau* starts being not small compared to the second derivative (i.e. g''.tau^2 not small) while I guess that for image intensities, spatial derivatives can be very high if no intensity smoothing scheme is used.
- note: the approximation tau* = sum_i tau_i (section 2.3) does not stand in the case of non-small deformations.
- still in section 2.3, I do not understand the statement "given that \nabla f is moderate": where does this property come from? or is "given" meant to be understood as "provided..." (i.e. under the assumption that...)?
- computational times could have been given (though I guess they are reasonable).
Other remarks:
- suggestion: I find the "slight abuse of notation" (of confusing the derivative with the gradient) a bit annoying and suggest to use a different symbol, such as \nabla g. This could be useful in particular in the following perspective:
- Mathematical side note: the "gradient" of a functional is not a uniquely-defined object in that it depends on the metric chosen in the tangent space. More clearly: the space of small deformations tau comes with an inner product (here L2, but one could choose another one), and the gradient \nabla g obtained depends on this inner product choice M, even though the derivative Dg is the same (they are related by Dg(tau) = < \nabla_M g | tau >_M for any tau). The choice of the metric can then be seen as a prior over desired gradient descent paths. In the paper, the deformation fields get smoothed by a Gaussian filter at some point (eq. 7), in order to be smoother: this can be interpreted as a prior (gradient descent paths should be made of smooth deformations) and as an associated inner product change (there do exist a metric M such that the gradient for that metric is \nabla_M g = S \nabla_L2 g). It is possible to favor other kind of deformations (not just smooth ones, but for instance rigid ones, etc. [and by the way this could make the link with "Manitest: Are classifiers really invariant?" by Fawzi and Frossard, BMVC 2015, who observe that a rigid motion can affect the classifier output]). If interested, you can check "Generalized Gradients: Priors on Minimization Flows" by Charpiat et al. for general inner products on deformations (in particular favoring rigid motion), and "Sobolev active contours" by Sundaramoorthi et al. for inner products more dedicated to smoothing (such as with the H1 norm).
- Note: about the remark in section 3.2: deformation-induced transformations are a subset of all possible transformations of the image (which are all representable with intensity changes), so it is expected that a training against attacks on the intensity performs better than a training against attacks on spatial deformations.