Modern machine learning algorithms are often susceptible to adversarial examples -maliciously crafted inputs that are undetectable by humans but that fool the algorithm into producing undesirable behavior. In this work, we show that adversarial examples exist in natural language classification: we formalize the notion of an adversarial example in this setting and describe algorithms that construct such examples. Adversarial perturbations can be crafted for a wide range of tasks -including spam filtering, fake news detection, and sentiment analysis -and affect different models -convolutional and recurrent neural networks as well as linear classifiers to a lesser degree. Constructing an adversarial example involves replacing 10-30% of words in a sentence with synonyms that don't change its meaning. Up to 90% of input examples admit adversarial perturbations; furthermore, these perturbations retain a degree of transferability across models. Our findings demonstrate the existence of vulnerabilities in machine learning systems and hint at limitations in our understanding of classification algorithms.
The paper shows that neural networks are sensitive to adversarial perturbation for a set of NLP text classifications. They propose constructing (model-dependent) adversarial examples by optimizing a function J (that doesn't seem defined in the paper) subject to a constraint c(x, x') < \gamma (i.e. that the original input and adversarial input should be similar)
c is composed of two constraints:
1. || v - v' ||_2 < \gamma_1, where v and v' are bag of embeddings for each input 
2. |log P(x') - log P(x)| < \gamma_2 where P is a language model
The authors then show that for 3 classification problems "Trec07p", "Yelp", and "News" and 4 models (Naive Bayes, LSTM, word CNNs, deep-char-CNNs) that the models that perform considerably worse on adversarial examples than on the test set.  Furthermore to test the validity of their adversarial examples, the authors show the following:
1. Humans achieve somewhat similar accuracy on the original adverarial examples (8 points higher on one dataset and 8 points lower on the other two)
2. Humans rate the writing quality of both the original and adversarial examples to be similar
3. The adversarial examples only somewhat transfer across models
My main questions/complaints/suggestions for the paper are:
-Novelty/Methodology. The paper has mediocre novelty given other similar papers recently. 
On question I have is about whether the generated examples are actually close to the original examples. The authors do show some examples that do look good, but do not provide any systematic study (e.g. via human annotation)
 This is a key challenge in NLP (as opposed to vision where the inputs are continuous so it is easy to perturb them and be reasonably sure that the image hasn't changed much). In NLP however, the words are discrete, and the authors measure the difference between an original example and the adversary only in continuous space which may not actually be a good measure of how different they are.
They do have some constraint that the fraction of changed words cannot differ by more than delta, but delta = 0.5 in the experiments, which is really large! (i.e. 50% of the words could be different according to Algorithm 1)
-Writing: the function J is never mathematically defined, neither is the function c (except that it is known to be composed of the semantic/syntactic similarity constraints).
The authors talk about "syntactic" similarity but then propose a language model constraint. I think is a better word is "fluency" constraint. 
The results in Table 3 and Table 6 seem different, shouldn't the diagonal of Table 6 line up with the results in Table 3?
-Experimental methodology (more of a question since authors are unclear): The authors write that "all adversarial examples are generated and evaluated on the test set".
There are many hyperparameters in the proposed authors' approach, are these also tuned on the test set? That is unfair to the base classifier. The adversarial model should be tuned on the validation set, and then the same model should be used to generate test set examples. (The authors can even show the validation adversarial accuracy to show how/if it deviates from the test accuracy)
-Lack of related work in NLP (see the anonymous comment for some examples). Even the related work in NLP that is cited e.g. Jia and Liang 2017 is obfuscated in the last page. The authors' introduction only refers to related works in vision/speech and ignores related NLP work.
Furthermore, adversarial perturbation is related to domain transfer  (since both involve shifts between the training and test distribution) and it is well known for instance that models that are trained on Wall Street Journal perform poorly on other domains.  See SJ Pan and Q Yang, A Survey on transfer learning, 2010, for some example references.