Unsupervised Pretraining for Sequence to Sequence Learning
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that the pretraining accelerates training and improves generalization of seq2seq models, achieving state-of-the-art results on the WMT English->German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves an improvement of 1.3 BLEU from the previous best models on both WMT*14 and WMT*15 English->German. On summarization, our method beats the supervised learning baseline.
Authors propose the use of layer-wise language model-like pretraining for encoder-decoder models. This allows to leverage separate source and target corpora (in unsupervised manner) without necessity of large amounts of parallel training corpora. The idea is in principle fairly simple, and rely on initial optimising both encoder and decoder with LSTM tasked to perform language modelling. The ideas are not new, and the paper is more like a successful compilation of several approaches that have been around for some time. The experimental validation, though, offers some interesting insights into importance of initialization, and the effectiveness of different initialisations approaches in enc-dec setting. The regulariser you propose to use on page 3, looks like typical multi-task objective function, especially it is used in an alternating manner would be interesting to see whether similar performance might have been obtained starting with this objective, from random initialisation. You should probably give credit for encoder-decoder like-RNN models published in 1990s. Minors: Pg. 2, Sec 2.1 2nd paragraph: can be different sizes -> can be of different sizes
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Unsupervised Pretraining for Sequence to Sequence Learning
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that the pretraining accelerates training and improves generalization of seq2seq models, achieving state-of-the-art results on the WMT English->German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves an improvement of 1.3 BLEU from the previous best models on both WMT*14 and WMT*15 English->German. On summarization, our method beats the supervised learning baseline.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Authors propose the use of layer-wise language model-like pretraining for encoder-decoder models. This allows to leverage separate source and target corpora (in unsupervised manner) without necessity of large amounts of parallel training corpora. The idea is in principle fairly simple, and rely on initial optimising both encoder and decoder with LSTM tasked to perform language modelling. The ideas are not new, and the paper is more like a successful compilation of several approaches that have been around for some time. The experimental validation, though, offers some interesting insights into importance of initialization, and the effectiveness of different initialisations approaches in enc-dec setting. The regulariser you propose to use on page 3, looks like typical multi-task objective function, especially it is used in an alternating manner would be interesting to see whether similar performance might have been obtained starting with this objective, from random initialisation. You should probably give credit for encoder-decoder like-RNN models published in 1990s. Minors: Pg. 2, Sec 2.1 2nd paragraph: can be different sizes -> can be of different sizes
assistant
Authors propose the use of layer-wise language model-like pretraining for encoder-decoder models. This allows to leverage separate source and target corpora (in unsupervised manner) without necessity of large amounts of parallel training corpora. The idea is in principle fairly simple, and rely on initial optimising both encoder and decoder with LSTM tasked to perform language modelling. The ideas are not new, and the paper is more like a successful compilation of several approaches that have been around for some time. The experimental validation, though, offers some interesting insights into importance of initialization, and the effectiveness of different initialisations approaches in enc-dec setting. The regulariser you propose to use on page 3, looks like typical multi-task objective function, especially it is used in an alternating manner would be interesting to see whether similar performance might have been obtained starting with this objective, from random initialisation. You should probably give credit for encoder-decoder like-RNN models published in 1990s. Minors: Pg. 2, Sec 2.1 2nd paragraph: can be different sizes -> can be of different sizes