Deep neural networks have been demonstrated to be vulnerable to adversarial attacks, where small perturbations intentionally added to the original inputs can fool the classifier. In this paper, we propose a defense method, Featurized Bidirectional Generative Adversarial Networks (FBGAN), to extract the semantic features of the input and filter the non-semantic perturbation. FBGAN is pre-trained on the clean dataset in an unsupervised manner, adversarially learning a bidirectional mapping between the high-dimensional data space and the low-dimensional semantic space; also mutual information is applied to disentangle the semantically meaningful features. After the bidirectional mapping, the adversarial data can be reconstructed to denoised data, which could be fed into any pre-trained classifier. We empirically show the quality of reconstruction images and the effectiveness of defense.
This work proposes to defend against adversarial examples by “denoising” the input image through an autoencoder (a BiGAN trained similar to InfoGAN) before classifying it with a standard CNN. The robustness of the model is evaluated on the L_infinity metric against FGSM and PGD.
My main criticism is as follows:
* Novelty: several defences are based on a similar principle and the contributions of this paper are unclear.
* Insufficient evidence: The evaluation is minimal (only FGSM and PGD, no decision-, transfer- or score-based attacks) and insufficient to support the claims.
* Gradient masking: There is at least one clear sign of gradient masking in the results (FGSM performing better than PGD).
### Novelty
The only prior work against which the paper compares is DefenseGAN. The only advantage over DefenseGAN being stated is performance (because no intermediate optimisation step is used). However, besides DefenseGAN there are several other defences that project the input onto the learned manifold of “natural” inputs, including (see prior work section in [1] for an up-to-date list):
* Adversarial Perturbation Elimination GAN
* Robust Manifold Defense
* PixelDefend (autoregressive probabilistic model)
* MagNets
### Insufficient evidence
The only attacks employed are two gradient-based techniques (FGSM and PGD). It is known that gradient-based techniques may suffer from gradient-masking (see also next point) and that the effectiveness of different attacks various greatly (which is why one should use many different attacks). Hence, a full evaluation of the model should include score-based and decision-based attacks.
### Gradient masking
In Figure 5 (b) the FGSM attack performs better than PGD for epsilon = 0.05 (66.4% vs 71.5%). PGD, however, should be strictly more powerful than FGSM if the gradients and the hyperparameters are ok.
Gradient masking is the primary reason for why 95% of all proposed defences turned out to be ineffective, and there are good reasons to believe that the same might affect this defence. The robustness evaluation has to be much more thorough and convincing before any substantiated claims about the bidirectional architecture proposed here can be derived. In addition, the difference to prior work has to be made much clearer.
[1] Schott et al. “Towards the first adversarially robust neural network model on MNIST”