Generating Wikipedia by Summarizing Long Sequences
We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.
 This paper considers the task of generating Wikipedia articles as a combination of extractive and abstractive multi-document summarization task where input is the content of reference articles listed in a Wikipedia page along with the content collected from Web search and output is the generated content for a target Wikipedia page. The authors at first reduce the input size by using various extractive strategies and then use the selected content as input to the abstractive stage where they leverage the Transformer architecture with interesting modifications like dropping the encoder and proposing alternate self-attention mechanisms like local and memory compressed attention. In general, the paper is well-written and the main ideas are clear. However, my main concern is the evaluation. It would have been nice to see how the proposed methods perform with respect to the existing neural abstractive summarization approaches. Although authors argue in Section 2.1 that existing neural approaches are applied to other kinds of datasets where the input/output size ratios are smaller, experiments could have been performed to show their impact. Furthermore, I really expected to see a comparison with Sauper & Barzilay (2009)*s non-neural extractive approach of Wikipedia article generation, which could certainly strengthen the technical merit of the paper. More importantly, it was not clear from the paper if there was a constraint on the output length when each model generated the Wikipedia content. For example, Figure 5-7 show variable sizes of the generated outputs. With a fixed reference/target Wikipedia article, if different models generate variable sizes of output, ROUGE evaluation could easily pose a bias on a longer output as it essentially counts overlaps between the system output and the reference. It would have been nice to know if the proposed attention mechanisms account for significantly better results than the T-ED and T-D architectures. Did you run any statistical significance test on the evaluation results? Authors claim that the proposed model can generate "fluent, coherent" output, however, no evaluation has been conducted to justify this claim. The human evaluation only compares two alternative models for preference, which is not enough to support this claim. I would suggest to carry out a DUC-style user evaluation (http://www-nlpir.nist.gov/projects/duc/duc2007/quality-questions.txt) methodology to really show that the proposed method works well for abstractive summarization. Does Figure 8 show an example input after the extractive stage or before? Please clarify. --------------- I have updated my scores as authors clarified most of my concerns.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Generating Wikipedia by Summarizing Long Sequences
We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper considers the task of generating Wikipedia articles as a combination of extractive and abstractive multi-document summarization task where input is the content of reference articles listed in a Wikipedia page along with the content collected from Web search and output is the generated content for a target Wikipedia page. The authors at first reduce the input size by using various extractive strategies and then use the selected content as input to the abstractive stage where they leverage the Transformer architecture with interesting modifications like dropping the encoder and proposing alternate self-attention mechanisms like local and memory compressed attention. In general, the paper is well-written and the main ideas are clear. However, my main concern is the evaluation. It would have been nice to see how the proposed methods perform with respect to the existing neural abstractive summarization approaches. Although authors argue in Section 2.1 that existing neural approaches are applied to other kinds of datasets where the input/output size ratios are smaller, experiments could have been performed to show their impact. Furthermore, I really expected to see a comparison with Sauper & Barzilay (2009)*s non-neural extractive approach of Wikipedia article generation, which could certainly strengthen the technical merit of the paper. More importantly, it was not clear from the paper if there was a constraint on the output length when each model generated the Wikipedia content. For example, Figure 5-7 show variable sizes of the generated outputs. With a fixed reference/target Wikipedia article, if different models generate variable sizes of output, ROUGE evaluation could easily pose a bias on a longer output as it essentially counts overlaps between the system output and the reference. It would have been nice to know if the proposed attention mechanisms account for significantly better results than the T-ED and T-D architectures. Did you run any statistical significance test on the evaluation results? Authors claim that the proposed model can generate "fluent, coherent" output, however, no evaluation has been conducted to justify this claim. The human evaluation only compares two alternative models for preference, which is not enough to support this claim. I would suggest to carry out a DUC-style user evaluation (http://www-nlpir.nist.gov/projects/duc/duc2007/quality-questions.txt) methodology to really show that the proposed method works well for abstractive summarization. Does Figure 8 show an example input after the extractive stage or before? Please clarify. --------------- I have updated my scores as authors clarified most of my concerns.
assistant
 This paper considers the task of generating Wikipedia articles as a combination of extractive and abstractive multi-document summarization task where input is the content of reference articles listed in a Wikipedia page along with the content collected from Web search and output is the generated content for a target Wikipedia page. The authors at first reduce the input size by using various extractive strategies and then use the selected content as input to the abstractive stage where they leverage the Transformer architecture with interesting modifications like dropping the encoder and proposing alternate self-attention mechanisms like local and memory compressed attention. In general, the paper is well-written and the main ideas are clear. However, my main concern is the evaluation. It would have been nice to see how the proposed methods perform with respect to the existing neural abstractive summarization approaches. Although authors argue in Section 2.1 that existing neural approaches are applied to other kinds of datasets where the input/output size ratios are smaller, experiments could have been performed to show their impact. Furthermore, I really expected to see a comparison with Sauper & Barzilay (2009)*s non-neural extractive approach of Wikipedia article generation, which could certainly strengthen the technical merit of the paper. More importantly, it was not clear from the paper if there was a constraint on the output length when each model generated the Wikipedia content. For example, Figure 5-7 show variable sizes of the generated outputs. With a fixed reference/target Wikipedia article, if different models generate variable sizes of output, ROUGE evaluation could easily pose a bias on a longer output as it essentially counts overlaps between the system output and the reference. It would have been nice to know if the proposed attention mechanisms account for significantly better results than the T-ED and T-D architectures. Did you run any statistical significance test on the evaluation results? Authors claim that the proposed model can generate "fluent, coherent" output, however, no evaluation has been conducted to justify this claim. The human evaluation only compares two alternative models for preference, which is not enough to support this claim. I would suggest to carry out a DUC-style user evaluation (http://www-nlpir.nist.gov/projects/duc/duc2007/quality-questions.txt) methodology to really show that the proposed method works well for abstractive summarization. Does Figure 8 show an example input after the extractive stage or before? Please clarify. --------------- I have updated my scores as authors clarified most of my concerns.