Real-world question answering systems often retrieve potentially relevant documents to a given question through a keyword search, followed by a machine reading comprehension (MRC) step to find the exact answer from them. In this process, it is essential to properly determine whether an answer to the question exists in a given document. This task often becomes complicated when the question involves multiple different conditions or requirements which are to be met in the answer. For example, in a question "What was the projection of sea level increases in the fourth assessment report?", the answer should properly satisfy several conditions, such as "increases" (but not decreases) and "fourth" (but not third). To address this, we propose a neural question requirement inspection model called NeurQuRI that extracts a list of conditions from the question, each of which should be satisfied by the candidate answer generated by an MRC model. To check whether each condition is met, we propose a novel, attention-based loss function. We evaluate our approach on SQuAD 2.0, NewsQA and MS MARCO datasets by integrating the proposed module with various MRC models, demonstrating the consistent performance improvements across a wide range of existing methods.
Paper Summary:
This paper proposes a neural question requirement inspection models called NeurQuRI.  It is different from existing answer verifiers in that NeurQuRI pinpoints where the mismatch occurs between the question and the candidate answer in unanswerable cases. Experiments with SQuAD 2.0 show the effectiveness of NeurQuRI.
Strengths:
—NeurQuRI improved the accuracy of three popular reading comprehension models, which are BERT, DocQA and QANet, on SQuAD 2.0.
—NeurQuRI requires 33M additional model parameters to check answerability of the candidate answer.  It is quite less than those of the previous answer verifier (150M) proposed by Hu et al. (2019).
—NeurQuRI can be easily extended to other question answering models and tasks.
Weaknesses:
—The authors do not experimentally compare their model to the answer verifier proposed by Hu et al. (2019). The base model of Hu et al. (2019), RMR, is different from the base models used in this paper, BERT, DocQA, and QANet.
—The authors use only one dataset (SQuAD 2.0) to evaluate their method.  NewsQA, MS MARCO, CoQA and QuAC also contain unanswerable questions.
Comments/Suggestions:
—I think that coverage mechanisms used in NMT (Tu et al., 2016) and summarization (See et al., 2017) should be included in the section of related work.
—The idea similar to NeurQuRI is used in a multi-hop QA model, proposed by Nishida et al. (2019), to find the evidence sentences that cover important information with respect to the question sentence.
—It is worthwhile to present the results of ablation tests with respect to the modified answerability label (described in Section 2.2).
Review Summary:
The paper is well motivated.  However, I think the authors need to compare NeurQuRI with the answer verifier proposed by Hu et al. (2019).  Also, I think this paper can benefit a lot with a more comprehensive analysis with other datasets such as CoQA.
References:
Hu et al.: Read + Verify: Machine Reading Comprehension with Unanswerable Questions. AAAI 2019: 6529-6537
Tu et al.: Modeling Coverage for Neural Machine Translation. ACL (1) 2016
See et al.: Get To The Point: Summarization with Pointer-Generator Networks. ACL (1) 2017: 1073-1083
Nishida et al.: Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction. ACL (1) 2019: 2335-2345
EDIT Nov. 20, 2019: 
I appreciate the authors' revision.  
The revision has satisfied my concerns, and I decided to increase the score of the paper (weak reject -> weak accept).