A known weakness of deep networks is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples. We propose Fortified Networks, a simple extension of existing networks, which "fortifies" the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well. Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good results due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space. We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner). We show that these improvements are achieved across a wide variety of hyperparameters.
This paper proposes a new defense to adversarial examples based on the 'fortification' of hidden layers using a denoising autoencoder. While building models that are robust to adversarial examples is an important and relevant research problem, I am not convinced by the evaluation of the defense.  Specific comments:
- The authors mostly evaluate their defense using FGSM (particularly on CIFAR). To truly establish the merit of a new defense, the authors must benchmark against state-of-the-art defenses such as PGD. It also seems like the epsilon values used for the PGD attacks are fairly small. The authors should report accuracies to a range of epsilon values for the PGD attack, as is standard.
- When the authors attack their models using PGD/FGSM, is this only on the classification loss or does this also include the denoising terms? Similar defenses which use denoisers have been broken once you run PGD on the full model [1].
- I do not really understand the motivation behind using an autoencoder here. Firstly, it is not clear that adversarial examples lie off the data manifold - they could form a very small set on the data manifold and thereby not affect standard generalization. Secondly, have the authors tried a simple regularization loss based on the error between hidden layer representations to a natural examples and the corresponding adversarial example? I think the authors must motivate the use of denoising autoencoders here by comparing to such a simple baseline.
General comment: The results hard to parse given the arrangement of figures and tables. Also, which approach are the authors denoting as ‘baseline adv. Train’ in the tables? 
Overall I feel like building defenses to adversarial examples is a challenging problem and the empirical investigation in this paper is not sufficient to illustrate any real progress on this front.
[1] Athalye, A., & Carlini, N. (2018). On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses. arXiv preprint arXiv:1804.03286.