Published as a conference paper at ICLR 2018 MONOTONIC CHUNKWISE ATTENTION
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-theart results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.
The paper proposes an extension to a previous monotonic attention model (Raffel et al 2017) to attend to a fixed-sized window up to the alignment position. Both the soft attention approximation used for training the monotonic attention model, and the online decoding algorithm is extended to the chunkwise model. In terms of the model this is a relatively small extention of Raffel et al 2017.
Results show that for online speech recognition the model matches the performance of an offline soft attention baseline, doing significantly better than the monotonic attention model. Is the offline attention baseline unidirectional or bidirectional? In case it is unidirectional it cannot really be claimed that the proposed model's performance is competitive with an offline model.
My concern with the statement that all hyper-parameters are kept the same as the monotonic model is that the improvement might partly be due to the increase in total number of parameters in the model. Especially given that w=2 works best for speech recognition, it not clear that the model extension is actually helping. My other concern is that in speech recognition the time-scale of the encoding is somewhat arbitrary, so possibly a similar effect could be obtained by doubling the time frame through the convolutional layer. While the empirical result is strong it is not clear that the proposed model is the best way to obtain the improvement.
For document summarization the paper presents a strong result for an online model, but the fact that it is still less accurate than the soft attention baseline makes it hard to see the real significance of this. If the contribution is in terms of speed (as shown with the synthetic benchmark in appendix B) more emphesis should be placed on this in the paper. 
Sentence summarization tasks do exhibit mostly monotonic alignment, and most previous models with monotonic structure were evaluated on that, so why not test that here?
I like the fact that the model is truely online, but that contribution was made by Raffel et al 2017, and this paper at best proposes a slightly better way to train and apply that model.
---
 The additional experiments in the new version gives stronger support in favour of the proposed model architecture (vs the effect of hyperparameter choices). While I'm still on the fence on whether this paper is strong enough to be accepted for ICLR, this version is certainly improves the quality of the paper.