Are Generative Classifiers More Robust to Adversarial Attacks?
There is a rising interest in studying the robustness of deep neural network classifiers against adversaries, with both advanced attack and defence techniques being actively developed. However, most recent work focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. In this paper, we propose and investigate the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs with low likelihood under the generative model. Experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and that the proposed detection methods are effective against many recently proposed attacks.
 This paper explores the potential of generative models for adversarial robustness. It presents some interesting and well formulated findings but is lacking in some meaningful ways. -It does a good job of summarizing the literature though it misses out on some very recent but relevant work -It introduces three detection methods and extensively evaluates them against several zero-knowledge attack types. Notably all the attack types are gradient based methods. -They present detailed performance metrics on the zero-knowledge attack, and introduce the perfect knowledge attack but do not present equivalently detailed results. -The results are presented in a confusing manor, and the paper is perhaps done a disservice by the inclusion of a very large number of tables both in the main text and the appendix without the necessary writing required to situate the reader. -The paper also briefly mentions the ongoing debate around "off-manifold" conjecture, only to them assume its correctness and based the entire work on the premise. ###Highlights *well written introduction and a good overview of generative modeling *very good visualizations and explanation of the detection methods *thorough results on zero-knowledge gradient based attack detection rates *interesting direction on adversarial examples ### Areas for improvement *more time should be spent on discussing the off manifold conjecture; the paper is based on the correctness of this conjecture though recent work has called its validity into question *only gradient based attacks are discussed though the paper title suggests robustness to all attacks *perfect-knowledge attacks should be explored in much more detail * the paper should streamline the results and findings, the large number of tables and results will confuse the reader and do not add to the argument *recent work on generative models for adversarial robustness should be discussed (https://arxiv.org/abs/1805.09190, https://arxiv.org/abs/1811.06969) *the two class cifar10 results are not particularly convincing as this is a substantial simplification of the cifar10 problem * the results on the full cifar10 data set make use a the extracted features of a pre-trained discriminative cifar10 network, this introduces the problem at layer activation adversarial attacks, but this is not mentioned in the paper (https://arxiv.org/pdf/1511.05122.pdf) Overall i think the work shows promise but is not yet ready for acceptance
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Are Generative Classifiers More Robust to Adversarial Attacks?
There is a rising interest in studying the robustness of deep neural network classifiers against adversaries, with both advanced attack and defence techniques being actively developed. However, most recent work focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. In this paper, we propose and investigate the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs with low likelihood under the generative model. Experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and that the proposed detection methods are effective against many recently proposed attacks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper explores the potential of generative models for adversarial robustness. It presents some interesting and well formulated findings but is lacking in some meaningful ways. -It does a good job of summarizing the literature though it misses out on some very recent but relevant work -It introduces three detection methods and extensively evaluates them against several zero-knowledge attack types. Notably all the attack types are gradient based methods. -They present detailed performance metrics on the zero-knowledge attack, and introduce the perfect knowledge attack but do not present equivalently detailed results. -The results are presented in a confusing manor, and the paper is perhaps done a disservice by the inclusion of a very large number of tables both in the main text and the appendix without the necessary writing required to situate the reader. -The paper also briefly mentions the ongoing debate around "off-manifold" conjecture, only to them assume its correctness and based the entire work on the premise. ###Highlights *well written introduction and a good overview of generative modeling *very good visualizations and explanation of the detection methods *thorough results on zero-knowledge gradient based attack detection rates *interesting direction on adversarial examples ### Areas for improvement *more time should be spent on discussing the off manifold conjecture; the paper is based on the correctness of this conjecture though recent work has called its validity into question *only gradient based attacks are discussed though the paper title suggests robustness to all attacks *perfect-knowledge attacks should be explored in much more detail * the paper should streamline the results and findings, the large number of tables and results will confuse the reader and do not add to the argument *recent work on generative models for adversarial robustness should be discussed (https://arxiv.org/abs/1805.09190, https://arxiv.org/abs/1811.06969) *the two class cifar10 results are not particularly convincing as this is a substantial simplification of the cifar10 problem * the results on the full cifar10 data set make use a the extracted features of a pre-trained discriminative cifar10 network, this introduces the problem at layer activation adversarial attacks, but this is not mentioned in the paper (https://arxiv.org/pdf/1511.05122.pdf) Overall i think the work shows promise but is not yet ready for acceptance
assistant
 This paper explores the potential of generative models for adversarial robustness. It presents some interesting and well formulated findings but is lacking in some meaningful ways. -It does a good job of summarizing the literature though it misses out on some very recent but relevant work -It introduces three detection methods and extensively evaluates them against several zero-knowledge attack types. Notably all the attack types are gradient based methods. -They present detailed performance metrics on the zero-knowledge attack, and introduce the perfect knowledge attack but do not present equivalently detailed results. -The results are presented in a confusing manor, and the paper is perhaps done a disservice by the inclusion of a very large number of tables both in the main text and the appendix without the necessary writing required to situate the reader. -The paper also briefly mentions the ongoing debate around "off-manifold" conjecture, only to them assume its correctness and based the entire work on the premise. ###Highlights *well written introduction and a good overview of generative modeling *very good visualizations and explanation of the detection methods *thorough results on zero-knowledge gradient based attack detection rates *interesting direction on adversarial examples ### Areas for improvement *more time should be spent on discussing the off manifold conjecture; the paper is based on the correctness of this conjecture though recent work has called its validity into question *only gradient based attacks are discussed though the paper title suggests robustness to all attacks *perfect-knowledge attacks should be explored in much more detail * the paper should streamline the results and findings, the large number of tables and results will confuse the reader and do not add to the argument *recent work on generative models for adversarial robustness should be discussed (https://arxiv.org/abs/1805.09190, https://arxiv.org/abs/1811.06969) *the two class cifar10 results are not particularly convincing as this is a substantial simplification of the cifar10 problem * the results on the full cifar10 data set make use a the extracted features of a pre-trained discriminative cifar10 network, this introduces the problem at layer activation adversarial attacks, but this is not mentioned in the paper (https://arxiv.org/pdf/1511.05122.pdf) Overall i think the work shows promise but is not yet ready for acceptance