Adversarial examples are crafted with imperceptible perturbations with the intent to fool neural networks. Against such attacks, adversarial training and its variants stand as the strongest defense to date. Previous studies have pointed out that robust models that have undergone adversarial training tend to produce more salient and interpretable Jacobian matrices than their non-robust counterparts. A natural question is whether a model trained with an objective to produce salient Jacobian can result in better robustness. This paper answers this question with affirmative empirical results. We propose Jacobian Adversarially Regularized Networks (JARN) as a method to optimize the saliency of a classifier's Jacobian by adversarially regularizing the model's Jacobian to resemble natural training images. Image classifiers trained with JARN show improved robust accuracy compared to standard models on the MNIST, SVHN and CIFAR-10 datasets, uncovering a new angle to boost robustness without using adversarial training examples.
This paper proposes a novel regularization strategy for improving the robustness of networks to adversarial noise. A term is added to the standard supervised cross-entropy loss that encourages the Jacobian of the network to itself be interpreted as a valid image. This "regularization" term is constructed by running the input-output Jacobian of the classification network through an "adapter network" and then in turn interpreting its output as a "generator" in a GAN setup. A separate discriminator network is training to distinguish real input images from these adapter-processed input-output Jacobians. The overall regularization is the standard minimax GAN loss applied to this generator/discriminator setup. The impetus for this stems from a previous observation that salient or interpretable input-output Jacobians naturally arise for networks that have undergone adversarial training to increase robustness.
Although this whole setup seems to be a little "Rube-Goldberg"-esque, I think there's some real sensible reasons for this sort of regularization to make intuitive sense. The input-output Jacobian characterizes how much the output (i.e. the logits) are affected by small changes to the input. The Jacobian, reinterpreted as living in the input image space (as the authors do), is a map of which input pixels have the strongest effect on the output of the network. If the Jacobian image looks like the underlying input image -- in particular, highlighting the labeled object -- this indicates that changing those pixels will result in the largest change on the network output. (This should be clear when looking at Figure 4 of the paper.) On the other hand, adversarial noise by definition leaves the underlying object alone (so that a human isn't aware of the perturbation) and modifies other pixels. Models that fall for such adversarial noise will not have salient Jacobians.
This is an amusing original idea, and I think this paper probably should be accepted to ICLR -- though I don't hold that position very strongly. However, I think the most interesting point is idea of Jacobian saliency, which is from prior work (Tsipras et al., 2018) that I haven't read. Therefore, I'm not sure how significant this paper is on it's own. Regardless, I would have liked to see more discussion in the paper of why Jacobian saliency should confer robustness (as I tried to do in the paragraph above), with perhaps some additional experiments designed around understanding whether this intuition (or something similar) is actually correct. There's some discussion of the theory behind the method in section 3.1, but it's not very intuitive to the situation at hand (non-linear neural networks), and I don't find it particularly informative.
Finally, some effort is spent arguing that this method is more computational efficient than adversarial training -- I wonder if that's still true when the all the complexity of GAN training is taken into account or how to consider that point when part of the conclusion is that their method is best when it is also combined with some amount of adversarial training.