MULTI-MENTION LEARNING FOR READING COMPREHENSION WITH NEURAL CASCADES
Reading comprehension is a challenging task, especially when executed across longer or across multiple evidence documents, where the answer is likely to reoccur. Existing neural architectures typically do not scale to the entire evidence, and hence, resort to selecting a single passage in the document (either via truncation or other means), and carefully searching for the answer within that passage. However, in some cases, this strategy can be suboptimal, since by focusing on a specific passage, it becomes difficult to leverage multiple mentions of the same answer throughout the document. In this work, we take a different approach by constructing lightweight models that are combined in a cascade to find the answer. Each submodel consists only of feed-forward networks equipped with an attention mechanism, making it trivially parallelizable. We show that our approach can scale to approximately an order of magnitude larger evidence documents and can aggregate information at the representation level from multiple mentions of each answer candidate across the document. Empirically, our approach achieves stateof-the-art performance on both the Wikipedia and web domains of the TriviaQA dataset, outperforming more complex, recurrent architectures.
The authors present a scalable model for questioning answering that is able to train on long documents. On the TriviaQA dataset, the proposed model achieves state of the art results on both domains (wikipedia and web). The formulation of the model is straight-forward, however I am skeptical about whether the results prove the premise of the paper (e.g. multi-mention reasoning is necessary). Furthermore, I am slightly unconvinced about the authors' claim of efficiency. Nevertheless, I think this work is important given its performance on the task.
1. Why is this model successful? Multi-mention reasoning or more document context?
I am not convinced of the necessity of multi-mention reasoning, which the authors use as motivation, as shown in the examples in the paper. For example, in Figure 1, the answer is solely obtained using the second last passage. The other mentions provide signal, but does not provide conclusive evidence. Perhaps I am mistaken, but it seems to me that the proposed model cannot seem to handle negation, can the authors confirm/deny this? I am also skeptical about the computation efficiency of a model that scores all spans in a document (which is O(N^2), where N is the document length). Can you show some analysis of your model results that confirm/deny this hypothesis?
2. Why is the computational complexity not a function of the number of spans?
It seems like the derivations presents several equations that score a given span. Perhaps I am mistaken, but there seems to be n^2 spans in the document that one has to score. Shouldn't the computational complexity then be at least O(n^2), which makes it actually much slower than, say, SQuAD models that do greedy decoding O(2n + nm)?
Some minor notes
- 3.3.1 seems like an attention computation in which the attention context over the question and span is computed using the question. Explicitly mentioning this may help the reading grasp the formulation.
- Same for 3.4, which seems like the biattention (Seo 2017) or coattention (Xiong 2017) from previous squad work.
- The sentence "We define ... to be the embeddings of the l words of the sentence that contains s." is not very clear. Do you mean that the sentence contains l words? It could be interpreted that the span has l words.
- There is a typo in your 3.7 "level 1 complexity": there is an extra O inside the big O notation.