Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms. Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples. However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images. While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear. In this paper, we study the distribution of softmax induced by stochastic transformations. We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction. Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images. With these observations, we propose a method to improve existing transformation-based defenses. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images. Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images. Our method is generic and can be integrated with existing transformation-based defenses.
===== Post Rebuttal ===== 
The authors addressed almost all of my comments. I think the revised paper is of higher quality and clarity as a result. Figure 3 is the main remaining concern. However,  given the listed strengths of the paper, I am happy to change my rating to weak accept .
===== Summary ===== 
One strategy for defense against adversarial attacks is to perturb the input sample multiple times and then aggregate their corresponding outputs into a single output. One recent variant of this method (Prakash et al. 2018) shows using this type of defense mechanism degrades the accuracy on clean (unperturbed) inputs. This paper tries to mitigate the accuracy drop on clean inputs. It does so by learning an additional distribution classifier which takes as input the distribution of perturbed samples’ outputs. The method is trained on MNIST, CIFAR10, CIFAR100 and using 4 different adversarial attacks. It shows significant improvement in several cases over standard input-perturbation methods.
===== Strengths and Weaknesses ===== 
+ The paper picks an important problem, proposes a simple technique and achieves significant improvements in certain cases.
+ The experiments are done on 3 datasets and 4 different adversarial attack techniques.
+ The final experiment on end-to-end attack is interesting.
Concerns regarding the main paper’s description
- From Figure 1, it seems that the distribution classifier does a binary classification for the distribution of each class separately. If this is the case, how is the final classification done (using the binomial distribution per class)? The figure suggests that the maximum probability of the binary classifications (bus for yellow class with 0.8 score) is taken. This would be strange since they are not optimized to compete with each other (e.g. as in a more proper cross entropy). In any case, the details of the distribution classifier is missing from the paper, and section 4 refers to the figure. The details should come in the main text using precise language and math formulation. 
- page 4 is very hard to follow since it tries to give a textual description to mathematical concepts. I think it is important to provide the mathematical definition such that 1) it is easier to follow 2) the concepts are unambiguously defined. This includes a definition of softmax distribution, joint distribution of output and transformation, marginalized distributions, clean intra-class distance, adversarial intra-class distance, and clean-adversarial inter-class distance.
- page 4: it seems it is implied in the definition of “clean” images that these images are classified “correctly” by the network. Otherwise, in Figure 2.a, the clean row is uninterpretable not knowing whether the clean image was classified correctly or not.
- page 4,5: what is a “transformation magnitude”? It could refer to both the number of transformations and also the number of perturbed pixels. The behavior in Figure 3 suggests that the latter should be the case, but it’s important to disambiguate.
- figure 3 is counterintuitive. The support (possible distributions) should normally live on a simplex (due the sum-to-one probability constraint). In figure 3, it seems there is support in the full unit hypercube. In particular, we can see density along the horizontal/vertical line of 1-density for one class (i.e. non-zero density for others in this case)
- page 6: “[...] to train a distribution classifier on the distributions obtained from clean images only [...]“. I cannot follow the argument at the end of the first paragraph that led to the proposed *training on clean images only*. I understand that from the qualitative demonstration of 8 different examples in Figure 4 the distributions of perturbed adversarial examples *might be* redundant to the perturbed wrongly-classified clean examples. However, even if it is redundant, why should it be harmful to train the distribution classifier with adversarial examples as well as clean examples?
Regarding Experiments:
- why is the evaluation done only based on accuracy of *correctly-classified* clean images and the recovery of *wrongly-classified* adversarial images? Couldn’t it be that the baseline (majority voting) does a better job than the distribution classifier on the “wrongly-classified” clean images and the “correctly-classified” adversarial images? 
- I think it's still interesting and informative to train the distribution classifier using both clean and perturbed inputs to compare the performance.
- page 6: why is there a need for applying a kernel density estimator followed by a discretization (e.g. for instance instead of a simple histogram)?
===== Final Decision ===== 
My current “weak reject” rating is primarily based on the unclarity of the main paper and secondary regarding the concerns for experiments. 
===== Points of Improvement ===== 
I believe the paper will become stronger if the proposal is described and motivated more formally.