In this paper, we investigate the problem of training neural machine translation (NMT) systems with more than 40 billion bilingual sentence pairs, which is orders of magnitude larger than the largest dataset to date. Several challenges distinguish this problem from previous NMT works such as the extreme noisiness of the data and prohibitively long training time. We propose practical solutions to handle these issues and demonstrate that large-scale pretraining will significantly improve NMT performances. By merely using vanilla transformer structures, we are able to push the BLEU score of WMT17 Chinese-English dataset to 32.3, with a significant performance boost of +3.2 over existing SOTA results, along with achieving SOTA results on WMT18 (+2.6) and WMT19 (+2.9).
The paper proposes an approach to train NMT models on extremely large parallel corpora. Because of the dataset size, training several epochs on the full dataset with a single model is too expensive. As a result, the dataset is split into several chunks, on which different models are trained independently. The different models are combined to form an ensemble model. Different strategies are proposed to split the dataset effectively. The resulting model achieves a performance of 32.3, outperforming the previous SOTA by 3.2 BLEU.
In Section 4.3, I'm curious about why only 6 layers are used in the encoder and decoder? Large scale pretraining like in BERT usually benefits from very large datasets, but also very large models. 6 layers seems very small given the size of the training set. Table 2 reports results with "Small" and "Large" models. What does this correspond to? Only Section 4.3 discusses the size of the model, but it does not mention different architectural choices.
In Section 5.1 the paper mentions "the single-model achieves a BLEU score of 29.7, already outperforming the current best system", but in Tables 1 and 2 it seems that the best score with single model is 28.7, not 29.7.
I feel that the results are a bit disappointing given the scale of the experiments. Table 2 suggests that a single model, even trained with 40B sentence pairs, does not outperform a single model trained with 20M sentence pairs as in "He et al., 2019", while being significantly more expensive to train. Also, the comparisons in Table 1 are done between single and ensemble models, which is not a fair comparison. The model with a BLEU score of 32.0 uses an ensemble of 10 models. What would be the performance of an ensemble of 10 models trained with the regular 20M parallel sentence pairs?
Also, did you try the approach of "Hassan et al (2018)" suggested in Section 4.1, where only in-domain sentences are selected? It is true that  "every time we encounter a new domain, we have to retrain the model", but I think this is still a more viable approach than pretraining on the full 40B sentences. Why not trying to train on the top-100M sentence pairs that are the most in-domain?
In the back-translation (BT) experiments, did you select 100M monolingual sentences randomly? If that is the case, this is expected to see a drop in performance, BT is usually a very effective, but not so much when the monolingual data is noisy or out of domain. Although it is critical to work with cleaned data in NLP (especially in the context of generation), dataset cleaning is not really addressed in the paper.
Overall, the experimental setup is impressive, but the improvements in terms of BLEU are relatively small, and the technical contributions seem quite thin to me for a ML conference. Moreover, the dataset used in the paper is not available to the research community, which prevents reproducibility. Also, as mentioned above, I think several important experiments are missing.