Text summarization aims to extract essential information from a piece of text and transform it into a concise version. Existing unsupervised abstractive summarization models use recurrent neural networks framework and ignore abundant unlabeled corpora resources. In order to address these issues, we propose TED, a transformer-based unsupervised summarization system with pretraining on largescale data. We first leverage the lead bias in news articles to pretrain the model on large-scale corpora. Then, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are abstractive and containing even higher proportions of novel tokens than those from supervised models.
Paper's Claims
The paper introduces a new unsupervised abstractive summarization approach called TED, using a Transformer encoder and decoder. Their main contributions are as follows:
1) Pretraining the encoder and decoder on news articles using the first beginning as the target summary.
2) Fine-tune on other datasets using so-called theme modeling, and separately a denoising loss.
3) TED's performance is claimed to significantly improve over GPT-2 while not being too far from the best unsupervised extractive summarization results.
Decision
Edit: After revisions and discussions, I recommend we accept this paper.
I am leaning towards accepting this paper mostly because of the contribution #1 above. Unsupervised learning using large quantities of text that have the property of being typically written in a style that synthesizes information in the first 1-3 sentences is a powerful idea. That the performance is improved compared to other unsupervised abstractive summarization confirms the importance of this approach.
However the importance of and justification for the fine-tuning steps are comparatively much more limited in my opinion. Also, some important details about the preprocessing for pre-training appear to be missing and they could be quite important. 
Detailed arguments for decision
I view this effort as aiming to reproduce the BERT approach in the context of abstractive summarization, which is a good idea. The most clever contribution is in leveraging un-labeled text using the first few sentences as the target summary for pretraining. The results of just this part are already beating previous approaches, while not requiring any in-domain data, which is quite powerful. 
However, some relatively important details regarding the methodology are omitted or only glossed over and it would greatly contribute to making this work more reproducible if the details were included (see my detailed notes below, notably regarding section 2.2). 
On the fine-tuning steps, I have several worries. First, why not fine-tune using supervised learning, as would be the analog to the BERT approach? Instead the authors go out of their way to do in-domain unsupervised learning, which provides a boost, yes, but still doesn't compare positively to extractive and/or supervised methods. Second, why not perform the theme modeling and denoising also -- or rather only -- on the unlabelled pretraining data? Why should it be done on the in-domain fine-tuning data instead (while not using the most valuable piece of in-domain information, namely the example summaries)? After all, it's a fully unsupervised approach and it can actually be performed on any text at all, whether a summary for it exists or not.
Again regarding the unsupervised approach, and to push the BERT analogy further, I'm wondering why not initialize the pretraining model with a BERT-style trained model? After all we could imagine building a system that adds more and more in-domain characteristics sequentially: first pretrain a BERT model, then fine-tune to summarization using what this paper calls pretraining, and then finally fine-tune again to a specific summarization domain. 
So, to conclude, I find that this paper goes in the right direction and introduces important ideas for pretraining and fine tuning unsupervised abstractive summarization models, but that some decisions about how to use the various ideas (theme and denoising but no supervised learning, in-domain vs during pretraining) have not been explored enough.
Extra notes
page 2, second line: pretrainleverages (typo)
section 2.1: fix first sentence to make it an actual sentence.
section 2.2: "we obtain three years of online new articles ... via a search engine" please be more specific about your methodology.
section 2.2: You should double check more throughly that there is no data leakage in test. There could be articles about the same exact events, years apart, for example. I doubt that this would be a big effect, but there are easily ways to find highly similar articles between the pretraining data and test data to make sure.
section 2.2: "Next we conduct following data cleaning" fix (typo?). Also that sentence probably belongs to the next paragraph.
section 2.2: Why did you pick the values that you did for the preprocessing heuristics (such as between 10-150 words, 150-1200 words, 3 sentences and not 2 or 1 or 4, the ratio 0.65, etc.)? Were other values tried?
section 2.2: You mention you end up with 21.4M articles. How many were there to start with? What's the filtering ratio?
section 2.2: You mention that you pick the model with the best ROUGE-L score on the validation set. How many models were there? What was different between them?
section 2.2, OOV Problem: the information in this whole subsection would fit better in 2.1 where 'tokens' are left generic without specifying which type of token you're considering.
Figure 1: I find the upper part of this figure very confusing. Why are there arrows going from the encoder/decoder to a summary, to theme loss, to article and back to encoder/decoder? It's important that the summary is never seen by the theme loss otherwise it's not unsupervised anymore, and I also don't see why the arrow would go through article *after* theme loss. I assume there must have been a mistake, please fix.
section 2.4: "the sequence is slightly shuffled by applying a permutation /sigma such that ..." The formula given here tells me that all token indices are shuffled with another token within a window k. That seems like a lot of moving around, and also depending on the implementation a token from the beginning could possibly end up at the very tail of the sentence by being picked iteratively again and again, thus falling outside the permutation distance k. Please provide more details on how this is done and a justification for why it was decided to do it this way.
Section 3.1: I'd like to know how long (preferably number of words, or at least number of wordpiece tokens) the summaries generated are. What determines how long they are, is it a fixed size, or the model decides to stop on his own (or when hitting some limit), or something else?
section 4.2: Do you have any idea why your unsupervised approach yields more novel n-grams than a the supervised model you compare against? This can be good as much as it can be bad, in that it could be going off-track. Yes humans have high novelty, but high novelty in itself isn't necessarily good. I don't find the argument that have more novel ngrams is intrinsically, necessarily good, compelling. If I'm wrong, then it would be nice to have better explanation in the paper.