Neural reading comprehension models have recently achieved impressive generalisation results, yet still perform poorly when given adversarially selected input. Most prior work has studied semantically invariant text perturbations which cause a model's prediction to change when it should not. In this work we focus on the complementary problem: excessive prediction undersensitivity where input text is meaningfully changed, and the model's prediction does not change when it should. We formulate a noisy adversarial attack which searches among semantic variations of comprehension questions for which a model still erroneously produces the same answer as the original question -and with an even higher probability. We show that -despite comprising unanswerable questions -SQuAD2.0 and NewsQA models are vulnerable to this attack and commit a substantial fraction of errors on adversarially generated questions. This indicates that current models-even where they can correctly predict the answer-rely on spurious surface patterns and are not necessarily aware of all information provided in a given comprehension question. Developing this further, we experiment with both data augmentation and adversarial training as defence strategies: both are able to substantially decrease a model's vulnerability to undersensitivity attacks on held out evaluation data. Finally, we demonstrate that adversarially robust models generalise better in a biased data setting with a train/evaluation distribution mismatch; they are less prone to overly rely on predictive cues only present in the training set and outperform a conventional model in the biased data setting by up to 11% F 1 .
As an extension of recent developments on adversarial attacks and defenses, this paper proposes a simple but effective technique called undersensitivity on machine comprehension task, where the input question is changed but the prediction does not change when it should be. They use two linguistically informed tricks; PoS and NER, to produce the perturbations. In addition to that, several techniques are developed for reducing the adversarial search spaces (Eq 1 and 3) and controlling the level of undersensitivity (Eq 2). 
In general, the paper is very well written and clear to read. The formulation of the problem is very straightforward, too. I enjoyed reading the overall paper, especially the experimental results, which provides lots of insights about the techniques. The proposed techniques are simple but they are well-executed in the experiment with reasonable justification. Please find my detailed comments below. 
Method. 
I appreciate the simplicity of the proposed models with clear motivations. Also, validation of the approaches is well-executed in the experiment.
I like the idea of linguistically-controlled perturbations using PoS and NER. However, there might be many other ways to control it: for example, parsing a sentence using a constituency parser and replacing each phrase with corresponding synonyms/antonyms using WordNet might be interesting. Or, based on the parse, negating the verb might be another way to try. I would expect more linguistically-informed perturbations like these, and I could find some of them from (Kang et al 2018, Ebrahimi et al., 2018). Also, adding a couple of them in the experiment might be interesting to understand the underlying logic of the perturbations. 
One major concern of the proposed approach is the sub-optimality by the pre-trained RC model. The undersensitivity (Eq 2) and adversarial search (Eq 3) are calculated by the probability scores predicted by the pre-trained models. This means that producing the new sample x’ is only based on the correctness of the pre-trained model on new samples generated, which sounds to be unreliable. Moreover, using the samples produced by this sub-optimal model may be very limited to produce samples under the sub-optimal space of questions. I wonder how the authors tackle this issue in the experiment. 
Experiment
Adversarial attacks should show how an existing system is fragile to be attacked, but at the same time augmenting or adversarially training with them needs to improve its generalization power of the system against the attacks. However, many of the adversarial attack papers mostly focus on the former but not the latter part. In this work, authors showed a result of adversarial training/augmentation but its generalization power on original task (i.e., HasAns case) was not that powerful. The unbiased data setup is interesting but still did not provide any insights about generalization from the adversaries. It would be more convincing to see how this generalization from adversarial attacks can take benefits from bit different tasks such as open-end reading comprehension as a perspective of data augmentation. 
I see no comparison with other attacking/defending methods in Tables 3 and 4. Adding the recent models (Ebrahimi et al., 2018, Wallance et al., 2019) may help understand how the proposed models are more effective than other techniques.