Published as a conference paper at ICLR 2018 QANET: COMBINING LOCAL CONVOLUTION WITH GLOBAL SELF-ATTENTION FOR READING COMPRE- HENSION
Current end-to-end machine reading and question answering (Q&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score 1 on the test set, which is significantly better than the best published F1 score of 81.8.
This paper proposes two contributions: first, applying CNNs+self-attention modules instead of LSTMs, which could result in significant speedup and good RC performance; second, enhancing the RC model training with passage paraphrases generated by a neural paraphrasing model, which could improve the RC performance marginally.
Firstly, I suggest the authors rewrite the end of the introduction. The current version tends to mix everything together and makes the misleading claim. When I read the paper, I thought the speeding up mechanism could give both speed up and performance boost, and lead to the 82.2 F1. But it turns out that the above improvements are achieved with at least three different ideas: (1) the CNN+self-attention module; (2) the entire model architecture design; and (3) the data augmentation method. 
Secondly, none of the above three ideas are well evaluated in terms of both speedup and RC performance, and I will comment in details as follows:
(1) The CNN+self-attention was mainly borrowing the idea from (Vaswani et al., 2017a) from NMT to RC. The novelty is limited but it is a good idea to speed up the RC models. However, as the authors hoped to claim that this module could contribute to both speedup and RC performance, it will be necessary to show the RC performance of the same model architecture, but replacing the CNNs with LSTMs. Only if the proposed architecture still gives better results, the claims in the introduction can be considered correct.
(2) I feel that the model design is the main reason for the good overall RC performance. However, in the paper there is no motivation about why the architecture was designed like this. Moreover, the whole model architecture is only evaluated on the SQuAD dataset. As a result, it is not convincing that the system design has good generalization. If in (1) it is observed that using LSTMs in the model instead of CNNs could give on par or better results, it will be necessary to test the proposed model architecture on multiple datasets, as well as conducting more ablation tests about the model architecture itself.
(3) I like the idea of data augmentation with paraphrasing. Currently, the improvement is only marginal, but there seems many other things to play with. For example, training NMT models with larger parallel corpora; training NMT models with different language pairs with English as the pivot; and better strategies to select the generated passages for data augmentation.
I am looking forward to the test performance of this work on SQuAD.