There is a rising interest in studying the robustness of deep neural network classifiers against adversaries, with both advanced attack and defence techniques being actively developed. However, most recent work focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. In this paper, we propose and investigate the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs with low likelihood under the generative model. Experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and that the proposed detection methods are effective against many recently proposed attacks.
In this work the authors propose and analyse generative models as defences against adversarial examples. In addition, three detection methods are introduced and an extension to deep features is suggested.
My main concerns are as follows (details below):
* Important prior work is not mentioned.
* Evaluation with direct attacks is only based on (very few) gradient-based techniques, many results are not reliable.
* There are signs of gradient masking (the common problem of robustness evaluation, in particular of only gradient-based techniques are used).
* The way detection rates are taken into account in the perfect knowledge scenario is confusing.
### Style
I like the idea of testing many different factorisation structures. However, that comes with the drawback that one needs to constantly check back what the abbreviations mean. Together with the three detection methods, the manuscript is quite confusing at times and should definitely be streamlined. One suggestion: remove the detection methods: I did not find any real conclusion about them but they are definitely side-tracking users away from the main results.
### Prior work
There is at least one closely related prior work not mentioned here: the analysis by synthesis model [1]. This model uses a variational auto encoder to learn class conditional distributions and shows high robustness on MNIST. Please make clear what your contribution is over this paper (other than testing several other factorisations).
### Evaluation problems
The robustness of models should be evaluated on different direct attacks ranging from gradient-based to score-based (e.g. NES [2]) to decision-based attacks [3]. Please take a look at [1] to see how a very extensive evaluation might look like. The results can be astonishingly different for different attacks, and so basing conclusion on only one or two attacks is dangerous (in particular if you only use gradient-based ones). One can also see that in your results, just check the variations you get between MIM and PGD. Also, rather then discussing (and showing in detail) results for individual attacks, the minimum adversarial distance for a given sample that can be found by any attack is much more comparable between models (which can also streamline the manuscript).
One can see signs of gradient masking in your results. For example, in Figure 3 the MIM attacks levels out at 20% for the DBX model. That can happen for iterative attacks if the gradient is masked. Similarly, in Figure 5 DBX-ZK (zero knowledge) is better in both accuracy and detection rate than DBX-PKK (which takes the KL-detection method into account and should thus either be better in accuracy or detection rate).
More generally, the perfect knowledge case, in which the attacker knows about the detector, should only count samples as adversarials which evade the detector and change the model decision. Thus, the detection rate should be zero. Otherwise I have no idea what trade-off between accuracy and detection rate you are actually targeting and how to compare the results.
Also, some intermediate results are conflicting with each other. E.g. in 4.1 you state “the usage of bottleneck is beneficial for better robustness”, but for L2 this is not true.
Also, I am not sure how conclusive the grey-box and black-box scenarios really are: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models.
### Minor
 * In 4.1 you say “as they fail to find near manifold adversarials”, but I don’t see how there can be L-infty adversarials on MNIST that are on-manifold (remember, MNIST pixel values are basically binary). Plus, in the zero-knowledge scenario there is nothing that enforces staying on this manifold.
 * Result presentation (Figure 3/5 & Table 1) is very different for different attack scenarios, which makes them hard to compare. Please unify.
 * Is the L2 distance you report in Table 1 the mean (or median) distance to adversarial examples. If so, GBZ (for which you state that C&W “failed on attacking” has actually a smaller mean adversarial distance than some other models (for which C&W is actually quite successful).
 * Grey-box scenario doesn’t make a lot of sense: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models. A similar confounder makes the black-box results difficult to interpret.
* Also, taking into account that the paper is two pages longer and thus calls for higher standards
Taken together, I find the general direction of the paper very interesting and I’d definitely encourage the authors to go further. At the current stage, however, I feel that (1) contributions are not sufficiently delineated to prior work, (2) the evaluation is not convincingly supporting the claims and that  (3) the manuscript needs to be streamlined (both in terms of text and figures).
[1] Schott et al. (2018) “Towards the first adversarially robust neural network model on MNIST” (https://arxiv.org/abs/1805.09190)
[2] Ilyas et al. (2018) “Black-box Adversarial Attacks with Limited Queries and Information” ( [https://arxiv.org/abs/1804.08598)](https://arxiv.org/abs/1804.08598)) 
[3] Brendel et al. (2018) “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models” (https://arxiv.org/abs/1712.04248)