Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that selftraining is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a "noisy" version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.
This paper presents a self-training approach for improving sequence-to-sequence tasks. As a preliminary experiment, this study randomly sampled 100k sentences from WMT 2014 English-German dataset (WMT100K, hereafter), trained a baseline (Transformer) model on WMT100K, and applied self-training methods on the remaining English sentences as the unlabeled monolingual data. After exploring different procedures for self-training, this study uses the fine-tuning strategy: train a model on the supervision data; build pseudo parallel data by predicting translations for all unlabeled data using the trained model; train a new model on the pseudo parallel data; and fine-tune the new model on the supervision data. This strategy alone gave a 3 points improvement of BLEU.
This paper hypothesized the reasons behind the performance improvements: beam-search decoding (+0.5 BLEU) and dropout (+1.2 BLEU). The authors argue that the beam-search decoding contributes partially to the performance gains, while the dropout accounts for the most. The authors infer that the dropout causes an implicit perturbation. Exploring different perturbation strategies, synthetic noise (e.g., input tokens are randomly dropped, masked, and shuffled) and paraphrase (round-trip translation, e.g., English-German-English), the authors reported no significant difference between these two strategies. Finally, this paper reports empirical results (on WMT 2014 English-German, FloRes English-Nepali, and English Gigaword (summarization task)) of self-training strategies presented in this paper. This paper concluded that self-training can be an effective method to improve generalization and that the noise injected during self-training plays a critical role for its success.
This paper is well structured and well written. It is interesting to see the improvements of the sequence-to-sequence tasks by using the self-training approach. This paper is interesting because it has a close connection with the back-translation approach that has been popular in recent years.
Although the hypotheses presented in this paper are interesting, they were not fully validated after all. The analyses on the loss functions, ablation tests, and experiments on the toy task can only bring indirect explanations about why we could observe the performance improvements. This impression is also demonstrated by the fact that this paper uses 'might' five times when explaining interpretations of the experimental results. Having said, I tend to agree that identifying the exact reason is difficult.
However, I have two other questions before recommending this paper: (1) whether the baseline model was trained sufficiently, and (2) whether this paper is about self-training strategies or regularization methods. 
(1) In order to accept the experimental results that the self-training approach can improve the performance, we need to make sure that the baseline model was trained sufficiently. However, the appendix explains, "we basically use the same learning rate schedule and label smoothing as in fairseq examples." I'm not sure whether this training procedure was fair among different models because the baseline model and self-supervised model received totally different number of training instances (100k vs 3.9M). This claim would be stronger if this paper could show an evidence that the baseline model was trained properly by, for example, explaining the stopping criteria for iterations, tuning hyper-parameters (e.g., learning rate) individually for the baseline and self-trained models, showing the mean and variance of BLEU scores with different initializations, and showing the training curve of the baseline model.
(2) We can view the self-training strategies presented in this paper as regularization methods. For this reason, I am wondering whether the self-training strategies presented in this paper are only for self-training or general to pure supervised setting. We can easily guess that the performance would drop if we removed the dropout from the baseline method. In contrast, I would like to see whether the synthetic noise could improve the performance of the baseline method alone, behaving as a regularization method. It would be useful to see the performance of the baseline method without the dropout and with the synthetic noise to highlight the effect of the presented strategies under the self-training scenario.
Minor comments:
It would be useful to see the number of unlabeled instances in Section 3.1.
Sectoin 3.2: "This is different from back-translation where new knowledge may originate from an additional backward translation model."
I'm not sure whether a backward translation model can introduce new knowledge because the supervision data are usually the same between forward and backward directions.
Section 3.3: "at test/decoding time the model does not use dropout"
To be precise, the weights are scaled by the dropout rate (p) in the decoding time.
Reference: Lample et al. 2018 should be replaced with another paper:
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato. Unsupervised Machine Translation Using Monolingual Corpora Only. ICLR 2018.