Machine comprehension of text is an important problem in natural language processing. A recently released dataset, the Stanford Question Answering Dataset (SQuAD), offers a large number of real questions and their answers created by humans through crowdsourcing. SQuAD provides a challenging testbed for evaluating machine comprehension algorithms, partly because compared with previous datasets, in SQuAD the answers do not come from a small set of candidate answers and they have variable lengths. We propose an end-to-end neural architecture for the task. The architecture is based on match-LSTM, a model we proposed previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al. (2015) to constrain the output tokens to be from the input sequences. We propose two ways of using Pointer Net for our tasks. Our experiments show that both of our two models substantially outperform the best results obtained by Rajpurkar et al. (2016) using logistic regression and manually crafted features. Besides, our boundary model also achieves the best performance on the MSMARCO dataset (Nguyen et al., 2016).
1 INTRODUCTION
Machine comprehension of text is one of the ultimate goals of natural language processing. While the ability of a machine to understand text can be assessed in many different ways, in recent years, several benchmark datasets have been created to focus on answering questions as a way to evaluate machine comprehension (Richardson et al., 2013; Hermann et al., 2015; Hill et al., 2016; Weston et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016). In this setup, typically the machine is first presented with a piece of text such as a news article or a story. The machine is then expected to answer one or multiple questions related to the text.
In most of the benchmark datasets, a question can be treated as a multiple choice question, whose correct answer is to be chosen from a set of provided candidate answers (Richardson et al., 2013; Hill et al., 2016). Presumably, questions with more given candidate answers are more challenging. The Stanford Question Answering Dataset (SQuAD) introduced recently by Rajpurkar et al. (2016) contains such more challenging questions whose correct answers can be any sequence of tokens from the given text. Moreover, unlike some other datasets whose questions and answers were created automatically in Cloze style (Hermann et al., 2015; Hill et al., 2016), the questions and answers in SQuAD were created by humans through crowdsourcing, which makes the dataset more realistic. Another real dataset, the Human-Generated MAchine Reading COmprehension dataset (MSMARCO) (Nguyen et al., 2016), provided a query together with several related documents collected from Bing Index. The answer to the query is generated by human and the answer words can not only come from the given text.
Given these advantages of the SQuAD and MSMARCO datasets, in this paper, we focus on these new datasets to study machine comprehension of text. A sample piece of text and three of its associated questions from SQuAD are shown in Table 1. Traditional solutions to this kind of question answering tasks rely on NLP pipelines that involve multiple steps of linguistic analyses and feature engineering, including syntactic parsing, named entity recognition, question classification, semantic parsing, etc. Recently, with the advances of applying neural network models in NLP, there has been
much interest in building end-to-end neural architectures for various NLP tasks, including several pieces of work on machine comprehension (Hermann et al., 2015; Hill et al., 2016; Yin et al., 2016; Kadlec et al., 2016; Cui et al., 2016). However, given the properties of previous machine comprehension datasets, existing end-to-end neural architectures for the task either rely on the candidate answers (Hill et al., 2016; Yin et al., 2016) or assume that the answer is a single token (Hermann et al., 2015; Kadlec et al., 2016; Cui et al., 2016), which make these methods unsuitable for the SQuAD/MSMARCO dataset. In this paper, we propose a new end-to-end neural architecture to address the machine comprehension problem as defined in the SQuAD/MSMARCO dataset. And for the MSMARCO dataset, we will only make use of the words in the given text to generate the answer.
Specifically, observing that in the SQuAD/MSMARCO dataset many questions could be entailed from some sentences in the original text, we adopt a match-LSTM model that we developed earlier for textual entailment (Wang & Jiang, 2016) as one layer of our model. We build a bi-directional match-LSTM on the given passage with attentions on the question for each word so that each position in the paragraph will have a hidden representation reflecting its relation to the question. Then we further adopt the Pointer Net (Ptr-Net) model developed by Vinyals et al. (2015) to select the words in these positions based on the hidden representations built by match-LSTM as an answer. We propose two ways to apply the Ptr-Net model for our task: a sequence model which selects the answer word by word, and a boundary model which only selects the start and end points of the answer span. Experiments on the SQuAD dataset show that our two models both outperform the best performance reported by Rajpurkar et al. (2016). Moreover, using an ensemble of several of our models, we can achieve very competitive performance on SQuAD. For the MSMARCO dataset, a real query based problem, our boundary model outperforms our sequence model with a big margin. It also outperforms the golden passage baseline.
Our contributions can be summarized as follows: (1) We propose two new end-to-end neural network models for machine comprehension, which combine match-LSTM and Ptr-Net to handle the special properties of the SQuAD dataset. To the best of our knowledge, we are the first to propose the boundary model which is more suitable to the SQuAD/MSMARCO tasks. And we are the first to integrate the attention-based word pair matching into machine comprehension tasks. (2) We have achieved the performance of an exact match score of 71.3% and an F1 score of 80.8% on the unseen SQuAD test dataset, which is much better than the feature-engineered solution (Rajpurkar et al., 2016). Our performance is also close to the state of the art on SQuAD, which is 74.8% in terms of exact match and 82.2% in terms of F1 collected from the SQuAD Leaderboard 1. Besides, our boundary model achieves the state-of-art performance on the MSMARCO dataset with BLUE1/2/3/4 40.7/33.9/30.6/28.7 and Rouge-L 37.3 2. (3) Our further visualization of the models reveals some useful insights of the attention mechanism for reasoning the questions. And we also show that the boundary model can overcome the early stop prediction problem in the sequence model. Besides, we also made our code available online 3.
1https://rajpurkar.github.io/SQuAD-explorer/ 2http://www.msmarco.org/leaders.aspx 3 https://github.com/shuohangwang/SeqMatchSeq
2 METHOD
In this section, we first briefly review match-LSTM and Pointer Net. These two pieces of existing work lay the foundation of our method. We then present our end-to-end neural architecture for machine comprehension.
2.1 MATCH-LSTM
In a recent work on learning natural language inference, we proposed a match-LSTM model for predicting textual entailment (Wang & Jiang, 2016). In textual entailment, two sentences are given where one is a premise and the other is a hypothesis. To predict whether the premise entails the hypothesis, the match-LSTM model goes through the tokens of the hypothesis sequentially. At each position of the hypothesis, attention mechanism is used to obtain a weighted vector representation of the premise. This weighted premise is then to be combined with a vector representation of the current token of the hypothesis and fed into an LSTM, which we call the match-LSTM. The matchLSTM essentially sequentially aggregates the matching of the attention-weighted premise to each token of the hypothesis and uses the aggregated matching result to make a final prediction.
2.2 POINTER NET
Vinyals et al. (2015) proposed a Pointer Network (Ptr-Net) model to solve a special kind of problems where we want to generate an output sequence whose tokens must come from the input sequence. Instead of picking an output token from a fixed vocabulary, Ptr-Net uses attention mechanism as a pointer to select a position from the input sequence as an output symbol. The pointer mechanism has inspired some recent work on language processing (Gu et al., 2016; Kadlec et al., 2016). Here we adopt Ptr-Net in order to construct answers using tokens from the input text.
2.3 OUR METHOD
Formally, the problem we are trying to solve can be formulated as follows. We are given a piece of text, which we refer to as a passage, and a question related to the passage. The passage is
represented by matrix P ∈ Rd×P , where P is the length (number of tokens) of the passage and d is the dimensionality of word embeddings. Similarly, the question is represented by matrix Q ∈ Rd×Q where Q is the length of the question. Our goal is to identify a subsequence from the passage as the answer to the question.
As pointed out earlier, since the output tokens are from the input, we would like to adopt the Pointer Net for this problem. A straightforward way of applying Ptr-Net here is to treat an answer as a sequence of tokens from the input passage but ignore the fact that these tokens are consecutive in the original passage, because Ptr-Net does not make the consecutivity assumption. Specifically, we represent the answer as a sequence of integers a = (a1, a2, . . .), where each ai is an integer between 1 and P , indicating a certain position in the passage.
Alternatively, if we want to ensure consecutivity, that is, if we want to ensure that we indeed select a subsequence from the passage as an answer, we can use the Ptr-Net to predict only the start and the end of an answer. In this case, the Ptr-Net only needs to select two tokens from the input passage, and all the tokens between these two tokens in the passage are treated as the answer. Specifically, we can represent the answer to be predicted as two integers a = (as, ae), where as an ae are integers between 1 and P .
We refer to the first setting above as a sequence model and the second setting above as a boundary model. For either model, we assume that a set of training examples in the form of triplets {(Pn,Qn,an)}Nn=1 are given. An overview of the two neural network models are shown in Figure 1. Both models consist of three layers: (1) An LSTM preprocessing layer that preprocesses the passage and the question using LSTMs. (2) A match-LSTM layer that tries to match the passage against the question. (3) An Answer Pointer (Ans-Ptr) layer that uses Ptr-Net to select a set of tokens from the passage as the answer. The difference between the two models only lies in the third layer.
LSTM Preprocessing Layer
The purpose for the LSTM preprocessing layer is to incorporate contextual information into the representation of each token in the passage and the question. We use a standard one-directional LSTM (Hochreiter & Schmidhuber, 1997) to process the passage 4 and the question separately, as shown below:
Hp = −−−→ LSTM(P), Hq = −−−→ LSTM(Q). (1)
The resulting matrices Hp ∈ Rl×P and Hq ∈ Rl×Q are hidden representations of the passage and the question, where l is the dimensionality of the hidden vectors. In other words, the ith column vector hpi (or h q i ) in H
p (or Hq) represents the ith token in the passage (or the question) together with some contextual information from the left.
Match-LSTM Layer
We apply the match-LSTM model (Wang & Jiang, 2016) proposed for textual entailment to our machine comprehension problem by treating the question as a premise and the passage as a hypothesis. The match-LSTM sequentially goes through the passage. At position i of the passage, it first uses the standard word-by-word attention mechanism to obtain attention weight vector −→α i ∈ R1×Q as follows:
−→ Gi = tanh(WqHq + (Wph p i + W r−→h ri−1 + bp)⊗ eQ), −→α i = softmax(wᵀ −→ Gi + b⊗ eQ), (2)
where Wq,Wp,Wr ∈ Rl×l, bp,w ∈ Rl×1 and b ∈ R are parameters to be learned, −→ Gi ∈ Rl×Q is the intermediate result, −→ h ri−1 ∈ Rl×1 is the hidden vector of the one-directional match-LSTM (to be explained below) at position i−1, and the outer product (·⊗eQ) produces a matrix or row vector by repeating the vector or scalar on the left for Q times.
Essentially, the resulting attention weight −→α i,j above indicates the degree of matching between the ith token in the passage with the jth token in the question. Next, we use the attention weight vector
4For the MSMARCO dataset, P is actually consisted of several unrelated documents. The previous state of pre-processing LSTM and match-LSTM to compute the first state of each document is set to zero.
−→α i to obtain a weighted version of the question and combine it with the current token of the passage to form a vector −→z i:
−→z i = [
hpi Hq−→α ᵀi
] , (3)
where Hq ∈ Rl×Q, −→α i ∈ R1×Q and hpi ∈ Rl×1. This vector −→z i is fed into a standard onedirectional LSTM to form our so-called match-LSTM: −→ h ri = −−−→ LSTM(−→z i, −→ h ri−1), (4) where −→ h ri ∈ Rl×1.
We further build a similar match-LSTM in the reverse direction. The purpose is to obtain a representation that encodes the contexts from both directions for each token in the passage.
Let −→ Hr ∈ Rl×P represent the hidden states [ −→ h r1, −→ h r2, . . . , −→ h rP ] and ←− Hr ∈ Rl×P represent [ ←− h r1, ←− h r2, . . . , ←− h rP ], the hidden states of match-LSTM in the reverse direction. We define H
r ∈ R2l×P as the concatenation of the two:
Hr = [−→ Hr ←− Hr ] . (5)
Answer Pointer Layer
The top layer, the Answer Pointer (Ans-Ptr) layer, is motivated by the Pointer Net introduced by Vinyals et al. (2015). This layer uses the sequence Hr as input. Recall that we have two different models: The sequence model produces a sequence of answer tokens but these tokens may not be consecutive in the original passage. The boundary model produces only the start token and the end token of the answer, and then all the tokens between these two in the original passage are considered to be the answer. We now explain the two models separately.
The Sequence Model: Recall that in the sequence model, the answer is represented by a sequence of integers a = (a1, a2, . . .) indicating the positions of the selected tokens in the original passage. The Ans-Ptr layer models the generation of these integers in a sequential manner. Because the length of an answer is not fixed, in order to stop generating answer tokens at certain point, we allow each ak to take up an integer value between 1 and P + 1, where P + 1 is a special value indicating the end of the answer. Once ak is set to be P + 1, the generation of the answer stops.
In order to generate the kth answer token indicated by ak, first, the attention mechanism is used again to obtain an attention weight vector βk ∈ R1×(P+1), where βk,j (1 ≤ j ≤ P + 1) is the probability of selecting the jth token from the passage as the kth token in the answer, and βk,(P+1) is the probability of stopping the answer generation at position k. βk is modeled as follows:
Fk = tanh(VH̃r + (Wahak−1 + b a)⊗ e(P+1)), (6) βk = softmax(vᵀFk + c⊗ e(P+1)), (7)
where H̃r ∈ R2l×(P+1) is the concatenation of Hr with a zero vector, defined as H̃r = [Hr;0], V ∈ Rl×2l,Wa ∈ Rl×l, ba,v ∈ Rl×1 and c ∈ R are parameters to be learned, Fk ∈ Rl×(P+1) is the intermediate result, (· ⊗ e(P+1)) follows the same definition as before, and hak−1 ∈ Rl×1 is the hidden vector at position k − 1 of an answer LSTM as defined below:
hak = −−−→ LSTM(H̃rβᵀk ,h a k−1). (8)
We can then model the probability of generating the answer sequence as p(a|Hr) = ∏ k p(ak|a1, a2, . . . , ak−1,Hr), (9)
and
p(ak = j|a1, a2, . . . , ak−1,Hr) = βk,j . (10)
To train the model, we minimize the following loss function based on the training examples:
− N∑
n=1
log p(an|Pn,Qn). (11)
The Boundary Model: The boundary model works in a way very similar to the sequence model above, except that instead of predicting a sequence of indices a1, a2, . . ., we only need to predict two indices as and ae. So the main difference from the sequence model above is that in the boundary model we do not need to add the zero padding to Hr, and the probability of generating an answer is simply modeled as
p(a|Hr) = p(as|Hr)p(ae|as,Hr). (12)
As this boundary model could point to a span covering too many tokens without any restriction, we try to manually limit the length of the predicted span and then search the span with the highest probability computed by p(as)× p(ae|as) as the answer.
3 EXPERIMENTS
In this section, we present our experiment results and perform some analyses to better understand how our models works.
3.1 DATA
We use the Stanford Question Answering Dataset (SQuAD) v1.1 and the human-generated Microsoft MAchine Reading COmprehension (MSMARCO) dataset v1.1 to conduct our experiments.
Passages in SQuAD come from 536 articles in Wikipedia covering a wide range of topics. Each passage is a single paragraph from a Wikipedia article, and each passage has around 5 questions associated with it. In total, there are 23,215 passages and 107,785 questions. The data has been split into a training set (with 87,599 question-answer pairs), a development set (with 10,570 questionanswer pairs) and a hidden test set.
For the MSMARCO dataset, the questions are user queries issued to the Bing search engine, the context passages are real Web documents and the answers are human-generated. We select the span that has the highest F1 score with the gold standard answer for training and only predict the span in the passages during evaluation. The data has been split into a training set (82326 pairs), a development set (10047 pairs) and a test set (9650 pairs).
3.2 EXPERIMENT SETTINGS
We first tokenize all the passages, questions and answers. We use word embeddings from GloVe (Pennington et al., 2014) to initialize the model. Words not found in GloVe are initialized as zero vectors. The word embeddings are not updated during the training of the model.
The dimensionality l of the hidden layers is set to be 150. We use ADAMAX (Kingma & Ba, 2015) with the coefficients β1 = 0.9 and β2 = 0.999 to optimize the model. Each update is computed through a minibatch of 30 instances. We do not use L2-regularization.
For the SQuAD dataset, the performance is measured by two metrics: percentage of exact match with the ground truth answers and word-level F1 score when comparing the tokens in the predicted answers with the tokens in the ground truth answers. Note that in the development set and the test set each question has around three ground truth answers. F1 scores with the best matching answers are used to compute the average F1 score. For the MSMARCO dataset, the metrics in the official tool of MSMARCO evaluation are BLEU-1/2/3/4 and Rouge-L, which are widely used in many domains.
3.3 RESULTS
The SQuAD and MSMARCO results of our models as well as the results of the baselines (Rajpurkar et al., 2016; Yu et al., 2016) are shown in Table 2. For the “LSTM with Ans-Ptr” models, they are
the experiments with the ablation of attention mechanism in match-LSTM. Specifically, we use the final representation of the question to replace the weighted sum of the question representations. For the MSMARCO dataset, as the context for each question is consisted of around 10 documents, the “Golden Passage” is to directly use the human labeled document which could answer the question as the prediction.
From the results in Table 2, we can see that the boundary model could clearly outperform the sequence model in a big margin on both datasets. We hypothesis that the sequence model is more likely to stop word generation earlier, and the boundary model can somehow overcome this problem. We have a statistical analysis on the answers generated by our sequence and boundary models shown in Table 3. We can see that the length of the answers generated by the sequence model is much shorter than the ground truth. Especially for the MSMARCO task where the answers are usually much longer, the sequence model could only generate 7 words on average, while the ground truth answers are 16 on average and the boundary model could generate nearly the same number of words with the ground truth. Several answers generated by our models are shown in Appendix A. From Table 2, we can also see that the performance gets poorer by removing the attention mechanism in match-LSTM, while for the MSMARCO dataset, the attention mechanism effects less, with no more than 2 percent reduction in BLEU and Rouge-L scores by attention mechanism ablation.
Based on the effectiveness of boundary pointer and match-LSTM, we conduct further exploration of the boundary model by adding element-wise comparison (hpi−H qαᵀi ) and (h p i H
qαᵀi ) into Eqn 3 in match-LSTM layer, adding 2 more bi-directional LSTM layers between match-LSTM and Ans-Ptr layers, and adding bi-directional Ans-Ptr. We show the ablation study of this further tuned model in Table 4. We can see that adding element-wise matching could make the biggest improvement for our boundary model. We also try to remove the phrase-level representation by removing the pre-process LSTM and using the word-level representations as the inputs of match-LSTM. Interestingly, we find the phrase-level representation effects little on the MSMARCO task.
Overall, we can see that both of our match-LSTM models have clearly outperformed the logistic regression model by Rajpurkar et al. (2016), which relies on carefully designed features. The improvement of our models over the logistic regression model shows that our end-to-end neural network models without much feature engineering are very effective on these tasks and datasets. Our boundary model also outperformed the DCR model (Yu et al., 2016), which maximizes the probability of the gold standard span from all the candidate spans through a neural network structure.
3.4 FURTHER ANALYSES
To better understand the strengths and weaknesses of our models, we perform some further analyses of the results below.
First, we suspect that longer answers are harder to predict. To verify this hypothesis, we analysed the performance in terms of both exact match and F1 score with respect to the answer length on the development set, as shown in Figure 2. For example, for questions whose answers contain more than 9 tokens, the F1 score of the boundary model drops to around 55% and the exact match score drops to only around 30%, compared to the F1 score and exact match score of close to 72% and 67%, respectively, for questions with single-token answers. And that supports our hypothesis.
Next, we analyze the performance of our models on different groups of questions, as shown in Figure 2. We use a crude way to split the questions into different groups based on a set of question words we have defined, including “what,” “how,” “who,” “when,” “which,” “where,” and “why.” These different question words roughly refer to questions with different types of answers. For example, “when” questions look for temporal expressions as answers, whereas “where” questions look for locations as answers. According to the performance on the development dataset, our models work the best for “when” questions. This may be because in this dataset temporal expressions are relatively easier to recognize. Other groups of questions whose answers are noun phrases, such as “what” questions, “which” questions and “where” questions, also get relatively better results. On the other hand, “why” questions are the hardest to answer. This is not surprising because the answers to “why” questions can be very diverse, and they are not restricted to any certain type of phrases.
Finally, we would like to check whether the attention mechanism used in the match-LSTM layer is effective in helping the model locate the answer. We show the attention weights α in Figure 3. In the figure the darker the color is the higher the weight is. We can see that some words have been well aligned based on the attention weights. For example, the word “German” in the passage is aligned well to the word “language” in the first question, and the model successfully predicts “German” as the answer to the question. For the question word “who” in the second question, the
word “teacher” actually receives relatively higher attention weight, and the model has predicted the phrase “Martin Sekulic” after that as the answer, which is correct. For the third question that starts with “why”, the attention weights are more evenly distributed and it is not clear which words have been aligned to “why”. For the last question, we can see that the word knowledge needed for generating the answer can also be detected by match-LSTM. For example, the words “European”, “Parliament”, “Council”, “European” and “Union” have higher attention weights on “governing” in the question. Even though our models can solve this type of questions, they are still not able to solve the questions that need multi-sentences reasoning. More answers generated by our models for the questions related to different kinds of reasoning are shown in Appendix B.
4 RELATED WORK
Machine comprehension of text has gained much attention in recent years, and increasingly researchers are building data-drive, end-to-end neural network models for the task. We will first review the recently released datasets and then some end-to-end models on this task.
4.1 DATASETS
A number of datasets for studying machine comprehension were created in Cloze style by removing a single token from a sentence in the original corpus, and the task is to predict the missing word. For example, Hermann et al. (2015) created questions in Cloze style from CNN and Daily Mail highlights. Hill et al. (2016) created the Children’s Book Test dataset, which is based on children’s stories. Cui et al. (2016) released two similar datasets in Chinese, the People Daily dataset and the Children’s Fairy Tale dataset.
Instead of creating questions in Cloze style, a number of other datasets rely on human annotators to create real questions. Richardson et al. (2013) created the well-known MCTest dataset and Tapaswi et al. (2016) created the MovieQA dataset. In these datasets, candidate answers are provided for each question. Similar to these two datasets, the SQuAD dataset (Rajpurkar et al., 2016) was also created by human annotators. Different from the previous two, however, the SQuAD dataset does not provide candidate answers, and thus all possible subsequences from the given passage have to be considered as candidate answers.
Besides the datasets above, there are also a few other datasets created for machine comprehension, such as WikiReading dataset (Hewlett et al., 2016) and bAbI dataset (Weston et al., 2016), but they are quite different from the datasets above in nature.
4.2 END-TO-END NEURAL NETWORK MODELS FOR MACHINE COMPREHENSION
There have been a number of studies proposing end-to-end neural network models for machine comprehension. A common approach is to use recurrent neural networks (RNNs) to process the given text and the question in order to predict or generate the answers (Hermann et al., 2015). Attention mechanism is also widely used on top of RNNs in order to match the question with the given passage (Hermann et al., 2015; Chen et al., 2016). Given that answers often come from the given passage, Pointer Network has been adopted in a few studies in order to copy tokens from the given passage as answers (Kadlec et al., 2016; Trischler et al., 2016). Compared with existing work, we use match-LSTM to match a question and a given passage, and we use Pointer Network in a different way such that we can generate answers that contain multiple tokens from the given passage.
Memory Networks (Weston et al., 2015) have also been applied to machine comprehension (Sukhbaatar et al., 2015; Kumar et al., 2016; Hill et al., 2016), but its scalability when applied to a large dataset is still an issue. In this work, we did not consider memory networks for the SQuAD/MSMARCO datasets.
The setting of visual question answering (Antol et al., 2015) is quite similar to machine comprehension, while their answers are usually very short. So the sequence order of the word-level attention representation used to align the figure and the question(Xu & Saenko, 2016; Fukui et al., 2016; Lu et al., 2016), are not used in VQA. While our model focus on the word-by-word attention and use
LSTM to concatenate the aligned pairs and that would be helpful to generate a longer sequence as answer.
5 CONCLUSIONS
In this paper, We developed two models for the machine comprehension problem defined in the Stanford Question Answering (SQuAD) and A Human-Generated MAchine Reading COmprehension (MSMARCO) datasets, both making use of match-LSTM and Pointer Network. Experiments on the SQuAD and MSMARCO datasets showed that our second model, the boundary model, could achieve a performance close to the state-of-the-art performance on the SQuAD dataset and achieved the state-of-the-art on the MSMARCO dataset. We also show the boundary model could overcome the early stop prediction problem of the sequence model.
In the future, we plan to look further into the different types of questions and focus on those questions which currently have low performance, such as the “why’ questions and multi-sentences related questions. We also plan to test how our models could be applied to other machine comprehension datasets.
6 ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centres in Singapore Funding Initiative.
We thank Pranav Rajpurkar for testing our model on the hidden test dataset and Percy Liang for helping us with the Dockerfile for Codalab.
A APPENDIX
We show the predictions our boundary and sequence models on two cases from two datasets in Table 5. It can be seen that the sequence model is more likely to predict a shorter sequence which is the problem of early stop prediction.
B APPENDIX
We show how four different models work on different type of questions in SQuAD dataset through Table 6. After the analysis of a hundred cases, we see that our models are not able to solve the questions that need multi-sentences reasoning. And the model without attention mechanism has less power to identify the important key word like the third case shown in Table 6.
Summary:
The paper presents a deep neural network for the task of machine comprehension on the SQuAD dataset. The proposed model is based on two previous works -- match-LSTM and Pointer Net. Match-LSTM produces attention over each word in the given question for each word in the given passage, and sequentially aggregates this matching of each word in the passage with the words in the question. The pointer net is used to generate the answer by either generating each word in the answer or by predicting the starting and ending tokens in the answer from the provided passage. The experimental results show that both the variants of the proposed model outperform the baseline presented in the SQuAD paper. The paper also shows some analysis of the results obtained such as variation of performance across answer lengths and question types.
Strengths:
1. A novel end-to-end model for the task of machine comprehension rather than using hand-crafted features.
2. Significant performance boost over the baseline presented in the SQuAD paper.
3. Some insightful analyses of the results such as performance is better when answers are short, "why" questions are difficult to answer.
Weaknesses/Questions/Suggestions:
1. The paper does not show quantitatively how much modelling attention in match-LSTM and answer pointer layer helps. So, it would be insightful if authors could compare the model performance with and without attention in match-LSTM, and with and without attention in answer pointer layer.
2. It would be good if the paper could provide some insights into why there is a huge performance gap between boundary model and sequence model in the answer pointer layer.
3. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required.
4. Could authors please explain why the activations resulting from {h^p}_i and {h^r}_{i-1} in G_i in equation 2 are being repeated across dimension of Q. Why not learn different activations for each dimension? 
5. I wonder why Bi-Ans-Ptr is not used in the ensemble model (last row in table 2) when it is shown that Bi-Ans-Ptr improves performance by 1.2% in F1.
6. Could authors please discuss and compare the DCR model (in table 2) in the paper in more detail?
Review Summary: The paper presents a reasonable end-to-end model for the task of machine comprehension on the SQuAD dataset, which outperforms the baseline model significantly. However, it would be good if more analyses / ablation studies / insights are included regarding -- how much attention helps, why is boundary model better than sequence model, how does the performance change when the reasoning required becomes difficult.