Adversarial examples are modified samples that preserve original image structures but deviate classifiers. Researchers have put efforts into developing methods for generating adversarial examples and finding out origins. Past research put much attention on decision boundary changes caused by these methods. This paper, in contrast, discusses the origin of adversarial examples from a more underlying knowledge representation point of view. Human beings can learn and classify prototypes as well as transformations of objects. While neural networks store learned knowledge in a more hybrid way of combining all prototypes and transformations as a whole distribution. Hybrid storage may lead to lower distances between different classes so that small modifications can mislead the classifier. A onestep distribution imitation method is designed to imitate distribution of the nearest different class neighbor. Experiments show that simply by imitating distributions from a training set without any knowledge of the classifier can still lead to obvious impacts on classification results from deep networks. It also implies that adversarial examples can be in more forms than small perturbations. Potential ways of alleviating adversarial examples are discussed from the representation point of view. The first path is to change the encoding of data sent to the training step. Training data that are more prototypical can help seize more robust and accurate structural knowledge. The second path requires constructing learning frameworks with improved representations.
Strength: 
The proposed approach is architecture-independent: the attack is constructed only from the dataset.
Weaknesses: 
Paper is not sufficiently well written for a venue like ICLR. 
Attack has very low success rate.
To the exception of Figures 4 and 5, many experiments are conducted on MNIST.
Feedback: 
Experimental results show that the attack is able to degrade a classifier’s performance by inserting perturbations that are computed on the data only. However, there are no baselines included to compare adversarial evasion rates achieved here to prior efforts. This makes it difficult to justify the fairly low success rate. In your rebuttal, could you clarify why baselines were not used to offer comparison points in the experiments?
Furthermore, strong conclusions are made from the results despite the lack of supporting evidence. For instance, on P10, the attack is said to “also explains why adversarial examples can be universal.”. However, not only does the attack achieve less success than universal adversarial examples would (so it cannot explain all of them) but also does it not share any characteristics with the way universal adversarial examples are crafted. Drawing such a strong conclusion thus most likely needs a lot more supporting evidence. 
Several directions would improve the content of the paper: 
* Complete existing experimental results by being more systematic. For instance, in Section 3.1, measurements are only performed on one pair of MNIST images. Without studying a significant portion of the test set of two datasets, it is very difficult to draw any conclusions from the limited evidence.
* Perform a human study to have all perturbed images labeled again. Indeed, because of the ways images are perturbed here, it is unclear how much perturbation can be added without changing the label of the resulting input. 
* Study how existing adversarial example techniques modify internal representations. This would help support conclusions made (e.g., about universal perturbations---see above). 
* Rewrite the related work section to scope it better: for instance, Sabour et al. in Adversarial Manipulation of Deep Representations and Wicker et al. in Feature-Guided Black-Box Safety Testing of Deep Neural Networks explore adversaries operating in the feature space. This will also help build better baselines for the evaluation.
Additional details: 
TLDR: typo in the first word
P1: The following definition of adversarial examples is a bit restrictive, because they are not necessarily limited to vision applications (e.g., they could be found for text or speech as well). “Adversarial examples are modified samples that preserve original image structures” 
P1: The following statement is a bit vague (what is obvious impact referring to?): “Experiments show that simply by imitating distributions from a training set without any knowledge of the classifier can still lead to obvious impacts on classification results from deep networks.”
P1: References do not typeset properly (the parentheses are missing: perhaps, the \citep{} command was not used?)
P2: What is the motivation for including references to prior work in the realm of image segmentation and more generally-speaking multi-camera settings in the related work section? 
P2: Typo in “ linear vibration”
P2: It remains difficult to make a conclusion about humans being robust to the perturbations introduced by adversarial examples. For instance, Elsayed et al. at NIPS 2018 found that time-constrained humans were also misled by adversarial examples crafted to evade ML classifiers: see Adversarial Examples that Fool both Computer Vision and Time-Limited Humans.
P2: Prior work suggests that the following conclusion is not entirely true: “Most of these kinds of examples are generated by carefully designed algorithms and procedures. This complexity to some extent shows that adversarial examples may only occupy a small percentage for total image space we can imagine with pixel representations.” For instance, Tramer et al. in ICLR 2018 found that adversarial subspaces were often large: “Ensemble Adversarial Training: Attacks and Defenses”.
P2: Others have looked at internal representations of adversarial examples so the following statement would be best softened: “To the best of our knowledge, this paper should be the first one that discusses adversarial examples from the internal knowledge representation point of view.”. See for instance, Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning by Papernot and McDaniel.
P3: Could you add pointers to support the description of human abstraction and sparsity? It reads a bit speculative as is, and adding some pointers would help relate the arguments made to relevant pointers for readers that are less familiar with this topic. 
P3: What is the motivation for including the discussion of computations performed by a neural network layer-by-layer in Section 2?
P4: Given that saliency maps can be manipulated easily and are only applicable locally, it appears that Figure 1 is too limited to serve as sufficient evidence for the following conclusion: “This, in some way, proves the point that the knowledge storage and representation of current neural networks are not exactly sparse prototype based.”
P5: The error rate reported on MNIST is quite low (45%). Even using the Fast Gradient Method, one should be able to have the error rate be as high as 90% on a standard CNN.
P7: Would you be able to provide references to backup the following statement? “This is a network structure that is very common.”
P10: How does the discussion in Section 4.2 relate to the attack described in the submission?