Lead bias is a common phenomenon in news summarization, where early parts of an article often contain the most salient information. While many algorithms exploit this fact in summary generation, it has a detrimental effect on teaching the model to discriminate and extract important information. We propose that the lead bias can be leveraged in a simple and effective way in our favor to pretrain abstractive news summarization models on large-scale unlabeled corpus: predicting the leading sentences using the rest of an article. Via careful data cleaning and filtering, our transformer-based pretrained model without any finetuning achieves remarkable results over various news summarization tasks. With further finetuning, our model outperforms many competitive baseline models. For example, the pretrained model without finetuning achieves state-of-the-art results on DUC-2003  and DUC-2004  datasets. The finetuned model obtains 3.2% higher ROUGE-1, 1.6% higher ROUGE-2 and 2.1% higher ROUGE-L scores than the best baseline model on XSum dataset. Human evaluations further show the effectiveness of our method.
This paper proposed an interesting idea on how we can leverage the lead bias in summarization datasets to pretrain abstractive news summarization models on large-scale unlabelled corpus in simple and effective way. 
For pre-training, they collected three years of online news articles data. Then, they take the top 3 sentences of the article as summary and the rest of the article as input document. For better choosing such article-summary pairs, they employ effective data cleaning and filtering process. Overall, they collected 21.4M articles for the pretraining. 
 
Overall, the pretrained model does decent on three summarization datasets without any fine-tuning. After fine-tuning the respective datasets, the gains seem significant. Especially on the XSum dataset, the improvements are remarkable. 
I believe that the idea is interesting but the experiments are incomplete and more investigation is required to make this paper stronger. Therefore I suggest to reject this paper. 
Arguments:
1) The important experiments that are missing in this paper are evaluating the proposed method on better human written summarization datasets -- DUC. The real world summarizations resemble more like the ones in the DUC dataset and it would be interesting to see if the transfer results of the pretrained model on the DUC datasets. The important question is to understand whether the pretrained model which took advantage of lead-bias could achieve good summaries on real summarization samples. This would also answer whether the pretrained just took advantage of the lead-bias issue of many large summarization datasets or does it really learn good summarization model. 
2) This paper has good idea but mainly missing ablation studies. For example, how does the proposed model do compared with GPT-2 in the fine-tuning setting, and how do these two models perform on the DUC datasets. 
3) During the dataset filtering/collection, a check on the quality of the filtering process by doing a small human study would have been a great addition. Also, instead of showing the output examples (which can go in the supplementary), human study comparing the quality of the pretrained model with fine-tuning and a baseline (can be from previous work) would have been better. 
Other minor questions
1) “we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest” -- is there any reason on fixing to these numbers? How did you make this decision ?
2) Even though the performance gains look visibly significant, I would suggest to report the statistical significance scores.