Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
Neural Machine Translation (NMT) has achieved remarkable progress with the quick evolvement of model structures. In this paper, we propose the concept of layer-wise coordination for NMT, which explicitly coordinates the learning of hidden representations of the encoder and decoder together layer by layer, gradually from low level to high level. Specifically, we design a layer-wise attention and mixed attention mechanism, and further share the parameters of each layer between the encoder and decoder to regularize and coordinate the learning. Experiments show that combined with the state-of-the-art Transformer model, layer-wise coordination achieves improvements on three IWSLT and two WMT translation tasks. More specifically, our method achieves 34.43 and 29.01 BLEU score on WMT16 English-Romanian and WMT14 English-German tasks, outperforming the Transformer baseline.
Update after Author Feedback:  Thanks for all the extra info about speed and alternative encoder-decoders.   Original Review:  This work builds directly off of Transformer networks. They make two contributions to that kind of architecture. The first is to suggest running the encoder and decoder stacks layer by layer instead of running the encoder stack and passing information to the decoder stack. The second is to actually tie the weights of the encoder and decoder.  +:   Shows that sharing the weights between the encoder and decoder of a transformer network is promising.This is surprising and worth knowing.  Running a decoder layer right after its corresponding encoder layer processes (rather than running the next encoder layer) is also an interesting augmentation to Transformer networks. Again surprising that this contributes to better performance. If this has been done in the past, I certainly haven’t seen it in recent work.  Adding the learned source and target tokens is an interesting method for ensuring that the model knows which language the embeddings are in.  Strong improvements on NMT over the baseline Transformer network.  -:  Giving all layers of the decoder access to corresponding layers in the encoder does not seem novel, at least in the way it is phrased at various points throughout the paper. Though the original Transformer paper (Vaswani et al 2017) did in fact use only the top level outputs of their encoder, Figure 1 of (Vaswani et al 2017) actually more closely depicts the corresponding access to encoder layers. Perhaps because of that depiction, some of the follow up work, for example Gu et al. 2017, did actually use layer wise attention without considering that a major contribution of their work.  This work only shows layer wise coordination for the Transformer, not for other kinds of encoder-decoders, so its not clear that this coordination is more generally useful. To make the broader claim, I think it is necessary to show this idea is empirically fruitful for at least one of CNNs or LSTMs as well. This is mentioned as future work, but as is, this comes off more as a tweak of the Transformer than evidence of a more generally applicable idea.  You have to have 14 layers instead of 6 by doing it this way. How does this increase in layers combined with layer-wise processing affect efficiency at train/inference time?   This paper did indeed shift the way that I would try doing seq2seq tasks in the future, and the results on NMT were strong. On first read, it seemed like an incremental change to Transformers, but after looking more closely, I do think it is worth it for the community to try out this different perspective on encoder-decoder architectures. Overall 7, confidence 4. The code is said to be open sourced after reviewing, so I'm very confident this will be reproducible.