Deep neural networks (DNNs) have achieved great success in various applications due to their strong expressive power. However, recent studies have shown that DNNs are vulnerable to adversarial examples which are manipulated instances targeting to mislead DNNs to make incorrect predictions. Currently, most such adversarial examples try to guarantee "subtle perturbation" by limiting the L p norm of the perturbation. In this paper, we aim to explore the impact of semantic manipulation on DNNs predictions by manipulating the semantic attributes of images and generate "unrestricted adversarial examples". In particular, we propose an algorithm SemanticAdv which leverages disentangled semantic factors to generate adversarial perturbation by altering controlled semantic attributes to fool the learner towards various "adversarial" targets. We conduct extensive experiments to show that the semantic based adversarial examples can not only fool different learning tasks such as face verification and landmark detection, but also achieve high targeted attack success rate against real-world black-box services such as Azure face verification service based on transferability. To further demonstrate the applicability of SemanticAdv beyond face recognition domain, we also generate semantic perturbations on street-view images. Such adversarial examples with controlled semantic manipulation can shed light on further understanding about vulnerabilities of DNNs as well as potential defensive approaches.
The authors describe a method for adversarially modifying a given (test) example that 1) still retains the correct label on the example, but 2) causes a model to make an incorrect prediction on it. The novelty of their proposed method is that their adversarial modifications are along a provided semantic axis (e.g., changing the color of someone's skin in a face recognition task) instead of the standard $L_p$ perturbations that the existing literature has focused on (e.g., making a very small change to each individual pixel). The adversarial examples that the authors construct, experimentally, are impressive and striking. I'd especially like to acknowledge the work that the authors put in to construct an anonymous link where they showcase results from their experiments. Thank you!
Overall, I think that this is interesting work that can help to broaden the study of adversarial examples and make them more applicable even in non-adversarial settings (e.g., by making models more robust to the changes in semantic attributes that the authors consider). There has been quite a bit of interest in the community in adversarial examples that are not just $L_p$ perturbations, and I believe that the authors' approach will encourage a good deal of follow-up research. 
However, my main concern with the paper is that in my opinion, it does not sufficiently address why it is important to generate adversarial examples in the way they do. For example:
1) Is the argument that this is a more powerful attack surface, so adversaries should take note (and defenders should figure out how to defend against this)? If that is the case, what is the attack model under which these attacks are realistic? For example, the original $L_\infty$ attacks are motivated in the sense that the adversarial examples are visually imperceptible, so they might not be noticed by the end-user. What is the equivalent argument for these semantic attacks?
2) Is the argument that these semantic attacks somehow capture a more realistic part of the data distribution over all natural images, and therefore it is good to have models that perform well on these semantic adversarial examples even if we're not concerned about an adversary (e.g., because the model might generalize better to other tasks or be more causally correct)? If that's the case, then I think this needs to be explored more. For example, what about the following straw man baseline: use a controllable semantic-attribute-based generator to generate semantically different images without any notion of an adversarial attack, and then do standard $L_p$ attacks on that generated image? How would that be better or worse than the proposed method?
3) Or is the argument that it is just good to be able to generate examples that models get wrong? If so, why, and why is this method better than other methods?
I think the paper would be significantly stronger if the importance and implications of their work were explicated along the above lines. For this reason, my current assessment is a weak reject, though I'd be open to changing this assessment.
=== Less critical comments, no need to respond or fix right away ===
While the overall concept and approach was clear, I generally found the notation and mathematical exposition difficult to follow. Please be more precise. Here is a non-exhaustive list of examples from section 3:
a) I'm not sure what's the difference between $x^\text{tgt}$ and $x^\text{adv}$, or between $x^\text{new}$ and  $x^*$. These seem to be used somewhat interchangeably?
b) Equation 3 is the central optimization problem in the paper, and should be written out explicitly using $\alpha$ as the optimization variable, instead of referring to equations 1 and 2 (in which $x^*$ doesn't even appear).
c) I didn't understand equation 4. What does assuming $M(x^\text{tgt}) = y^\text{tgt}$ mean? What happens when that is not true?
d) Equation 5: Why is $y$ in the right hand side by not in the left?
e) Equation 6: $L_\text{smooth}$ is missing an argument.