Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and selfmatching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both stateof-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.
This paper proposes a new machine comprehension model, which integrates several contributions like different embeddings for gate function and passage representation function, self-attention layers and highway network based fusion layers. The proposed method was evaluated on the SQuAD dataset only, and marginal improvement was observed compared to the baselines.
(1) One concern I have for this paper is about the evaluation. The paper only evaluates the proposed method on the SQuAD data with systems submitted in July 2017, and the improvement is not very large. As a result, the results are not suggesting significance or generalizability of the proposed method.
(2) The paper gives some ablation tests like reducing the number of layers and removing the gate-specific question embedding, which help a lot for understanding how the proposed methods contribute to the improvement. However, the results show that the deeper self-attention layers are indeed useful (but still not improving a lot, about 0.7-0.8%). The other proposed components contribute less significant. As a result, I suggest the authors add more ablation tests regarding (1) replacing the outer-fusion with simple concatenation (it should work for two attention layers); (2) removing the inner-fusion layer and only use the final layer's output, and using residual connections (like many NLP papers did) instead of the more complicated GRU stuff.
(3) Regarding the ablation in Table 2, my first concern is that the improvement seems small (~0.5%). As a result, I am wondering whether this separated question embedding really brings new information, or the similar improvement can be achieved by increasing the size of LSTM layers. For example, if we use the single shared question embeddings, but increase the size from 128 to some larger number like 192, can we observe similar improvement. I suggest the authors try this experiment as well and I hope the answer is no, as separated input embeddings for gate functions was verified to be useful in some "old" works with syntactic features as gate values, like "Semantic frame identification with distributed word representations" and "Learning composition models for phrase embeddings" etc.
(4) Please specify which version of the SQuAD leaderboard is used in Table 3. Is it a snapshot of the Jul 14 one? Because this paper is not comparing to the state-of-the-art, no specification of the leaderboard version may confuse the other reviewers and readers. By the way, it will be better to compare to the snapshot of Oct 2017 as well, indicating the position of this work during the submission deadline.
Minor issues:
(1) There are typos in Figure 1 regarding the notations of Question Features and Passage Features.
(2) In Figure 1, I suggest adding an "N \times" symbol to the left of the Q-P Attention Layer and remove the current list of such layers, in order to be consistent to the other parts of the figure.
(3) What is the relation between the "PhaseCond, QPAtt+" in Table 2 and the "PhaseCond" in Table 3? I was assuming that those are the same system but did not see the numbers match each other.