Neural sequence-to-sequence models are at the basis of state-of-the-art solutions for sequential prediction problems such as machine translation and speech recognition. The models typically assume that the entire input is available when starting target generation. In some applications, however, it is desirable to start the decoding process before the entire input is available, e.g. to reduce the latency in automatic speech recognition. We consider state-of-the-art "wait-k" decoders, that first read k tokens from the source and then alternate between reading tokens from the input and writing to the output. We investigate the sensitivity of such models to the value of k that is used during training and when deploying the model, and the effect of updating the hidden states in transformer models as new source tokens are read. We experiment with German-English translation on the IWSLT14 dataset and the larger WMT15 dataset. Our results significantly improve over earlier state-of-the-art results for German-English translation on the WMT15 dataset across different latency levels.
This paper extends the idea of prefix-to-prefix in STACL and proposes two different variations. The authors did some interesting experiments between caching and updating decoder.
My questions are as follows:
1) in section 3.1.2, the authors mentioned two adaptations. Is the proposed AR encoder uni-directional? If the AR encoder is uni-directional, then I would be surprised that the uni-directional encoder outperforms the bi-directional encoder in the original wait-k model. For the second bullet, I think the original wait-k also did the same thing(they mentioned this in the paper clearly). So there is nothing new about bullet 2. 
2) the idea mentioned in fig 1 is very similar to [1]. I suggest the authors compare with the aforementioned methods. 
3) updating the hidden state of the decoder introduces more complexity during the inference time. I recommend the authors to perform some analysis about decoding time with CPU and GPU.
4) it is also interesting to show more comparison between different models' training time with the original STACL. 
[1] Zheng et al. "Simultaneous Translation with Flexible Policy via Restricted Imitation Learning" ACL 2019