COARSE-GRAIN FINE-GRAIN COATTENTION NET- WORK FOR MULTI-EVIDENCE QUESTION ANSWERING
End-to-end neural models have made significant progress in question answering, however recent studies show that these models implicitly assume that the answer and evidence appear close together in a single document. In this work, we propose the Coarse-grain Fine-grain Coattention Network (CFC), a new question answering model that combines information from evidence across multiple documents. The CFC consists of a coarse-grain module that interprets documents with respect to the query then finds a relevant answer, and a fine-grain module which scores each candidate answer by comparing its occurrences across all of the documents with the query. We design these modules using hierarchies of coattention and selfattention, which learn to emphasize different parts of the input. On the Qangaroo WikiHop multi-evidence question answering task, the CFC obtains a new stateof-the-art result of 70.6% on the blind test set, outperforming the previous best by 3% accuracy despite not using pretrained contextual encoders.
This paper proposes an interesting coarse-grain fine-grain coattention network architecture to address multi-evidence question answering and achieves the new state-of-the-art results on the Qangaroo WikiHop dataset.  The main idea is to divide the task across the coarse-grain and fine-grain modules in a complimentary manner such that the coarse-grain module learns from efficient modeling of support documents and the query whereas the fine-grain module learns from associations of candidate mentions in the support documents with the query. 
The major strength of the model is observed with learning effective representations of larger numbers of long support documents and the state-of-the-art results are achieved without the use of pretrained contextualized embeddings. The main novelty lies in how the coattention and self-attention strategies are combined hierarchically to learn relevant representations in a complimentary fashion (rather than serial). Overall, the paper is very well-written and presents solid results with meaningful ablation study, quantitative and qualitative analyses. I have a few comments/suggestions:
- It would be interesting to see how the inclusion of pretrained contextualized embeddings such as ELMo, ULMFit, BERT would help the current model. 
- "This is likely because coreference resolution captures intra-document and inter-document dependencies more accurately than hierarchical attention." --> Please clarify why this is the case.
- "We hypothesize that ways to reduce this type of error include using more robust pretrained contextual encoders (McCann et al., 2017; Peters et al., 2018) and coreference resolution." --> I agree; also, it would be worth considering some commonsense knowledge to alleviate this issue, because the fact that Scotland is a part of UK and has a border with England should be learned. Here is a relevant work: "Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge" by Mihaylov and Frank, 2018.
- "The second type (28% of errors) results from questions that are not answerable. For example, the support documents do not provide the narrative location of the play “The Beloved Vagabond” for the query narrative location the beloved vagabond." --> It would be great if you could release the set of unanswerable questions for the community.
- Please include the memory network-based QA works in the related work section because they involve some forms of reasoning. Also, I would suggest to cover the query-focused multi-document summarization area in the related work section because they also require evidence synthesis from multiple documents to address a query. It would be very interesting if authors can apply their model for the query-focused multi-document summarization task as well, as this would further validate the effectiveness of the proposed architecture for reasoning across multiple documents.