Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE, SQuAD, SuperGLUE and XNLI. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
This paper presents a replication study of BERT pretraining and carefully measures the impact of many key hyperparameters and training data size. It shows that BERT was significantly undertrained and propose an improved training recipe called RoBERTa. The key ideas are: (i) training longer with bigger batches over more data, (ii) removing NSP, (iii) training over long sequences, and (iv) dynamically changing the masking pattern. The proposed RoBERTa achieves/matches state-of-the-art performance on many standard NLU downstream tasks. 
The in-depth experimental analysis of the BERT pretraining process in this paper answers many open questions (e.g., the usefulness of NSP objective) and also provide some guidance in how to effectively tweak the performance of pretrained model (e.g., large batch size). It also further demonstrates that the BERT model, once fully tuned, could achieve SOTA/competitive performance compared to the recent new models (e.g., XLNet). The main weakness of the paper is that it is mainly based on further tuning the existing BERT model and lacks novel contribution in model architecture. However, the BERT analysis results provided in this paper should also be valuable to the community. 
Questions & Comments:
•	It is stated that the performance is sensitive to epsilon in AdamW. This reminds us of the sensitivity of BERT pretraining to the optimizers. Since one of the main contributions of this paper is the analysis of the BERT pretraining process, more experimental analysis on the optimizer should also be included.
•	It is stated that (page 7) the submission to GLUE leaderboard uses only single-task finetuning. Is there any special reason for restraining it to single-task finetuning if earlier results demonstrates multi-task finetuning is better? Of course, it is valuable to see the great performance achieved by single-task finetuning for RoBERTa. But there should be no reason that it is restricted to be so. An additional experimental results with multi-task finetuning should also be added.