Densely Connected Attention Propagation for Reading Comprehension
We propose DECAPROP (Densely Connected Attention Propagation), a new densely connected neural architecture for reading comprehension (RC). There are two distinct characteristics of our model. Firstly, our model densely connects all pairwise layers of the network, modeling relationships between passage and query across all hierarchical levels. Secondly, the dense connectors in our network are learned via attention instead of standard residual skip-connectors. To this end, we propose novel Bidirectional Attention Connectors (BAC) for efficiently forging connections throughout the network. We conduct extensive experiments on four challenging RC benchmarks. Our proposed approach achieves state-of-the-art results on all four, outperforming existing baselines by up to 2.6% − 14.2% in absolute F1 score.
This paper proposes a new neural architecture for reading comprehension.  Compared to many other existing neural architectures, this model 1) densely connects all pairs of passage layers and question layers for encoding; 2) uses a component called Bidirectional Attention Connectors (BAC)  for connecting any P-layer and Q-layer which employs an FM layer on top of commonly used bi-directional attention.  The proposed architecture has been evaluated on four reading comprehension datasets and demonstrates strong empirical results.  Overall, although the proposed ideas could be potentially interesting, I think the presented results (in the current presentation format) are not convincing enough.   - I am not sure how the four datasets were chosen, but this model is not evaluated on the most competitive datasets: SQuAD and (arguably) TriviaQA. This makes the results less convincing.  - In general, I find it is difficult to understand the influence of the proposed components in this paper. According to the ablation study in Table 6, I don’t see any of the components contributed that much to the overall performance and whether the results can generalize to other datasets.  I think it is necessary to have more ablation studies on other datasets and apparently, this model has also improved more on other datasets. Gated attention is not novel for this paper. How useful are these dense connections are indeed? How useful is the FM layer (if we just replace function G with a feedforward network)? I think it is important to highlight the new components of this paper and their contributions to a reading comprehension system (on various datasets).  - I think the paper could have done a better job explaining the intuitions of the proposed solutions. The paper made a few claims in the introduction part such as “densely connected by attention” and “compress the attention outputs so that they can be small enough to propagate” but it is unclear how they are reflected in the model design.  - The notions in this paper are also confusing.      - Line 76 and equation (3): how are b_i and p_i defined?     - How are Z* used/propagated in the following layers?  ========= I have carefully read all other reviews and the authors' rebuttal. Given the authors provided results on SQuAD and comparisons of FM vs feedforward networks, I increased my score from 5 to 6.