Teaching Machines to Read and Comprehend
Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.
This paper focuses on the reading comprehension task. It proposes to construct a large dataset using news stories and highlights from CNN and Daily Mail. Reading comprehension is formulated as a Cloze test, where a named entity in the news highlight is replaced by a placeholder. The system is expected to predict the missing entity (answer) based on the entire news document (document) and highlight (query). The paper explores several approaches to address the problem. The baselines are based on entity frequency, frame semantic parsing, and word distance. A set of neural models, including a multi-layer LSTM and LSTMs with attention mechanism, are compared against the baselines. Results show that adding the attention mechanism helps, and the 'impatient reader' model yields the best performance.The paper is well structured in general. The proposed approach for constructing supervised reading comprehension dataset using news documents and highlights seems interesting. I have a few concerns with the paper, they are listed below.It seems the baselines used in the paper are pretty weak. The frame semantic parsing and word distance benchmarks are both unsupervised and they seem to be ad-hoc. Although the frame semantic parser is used to retrieve predicate/argument triples, only heuristic rules are used to combine the triples for answer prediction. I feel the state-of-the-art NLP approaches could do much better than that. Perhaps even a supervised extension would generate better results.I'd also be curious to know the difficulty of the proposed Cloze test. In particular, how well human subjects would perform on the anonymized test. I suspect entities in the news highlight occur frequently in the document. It would be helpful to provide some coverage statistics. For example, the percentage of gold-standard answers that are covered in the top-3 or top-5 most frequent entities. If the coverage is high, perhaps the task is not as difficult as it may seem.Albeit the good performance, I'm not fully convinced that LSTM would be the method-of-choicefor this task. It seems adding the attention mechanism certainly works, since it helps identify the passages that are related to the answer. However, I'm a bit surprised to see that without the attention mechanism, the single-layer LSTM model yields much worse performance ('uniform attention' in Table 5). I'm also not sure why 'impatient reader' performs well. Some explanations on the advantage of this model would be helpful.The question categorization in Section 2.1 seems unclear. The definition of different categories needs clarification. Providing example queries may help. If space is a concern, it seems okay to completely remove this section.