Read, Highlight and Summarize: A Hierarchical Neural Semantic Encoder-based Approach
Traditional sequence-to-sequence (seq2seq) models and other variations of the attention-mechanism such as hierarchical attention have been applied to the text summarization problem. Though there is a hierarchy in the way humans use language by forming paragraphs from sentences and sentences from words, hierarchical models have usually not worked that much better than their traditional seq2seq counterparts. This effect is mainly because either the hierarchical attention mechanisms are too sparse using hard attention or noisy using soft attention. In this paper, we propose a method based on extracting the highlights of a document; a key concept that is conveyed in a few sentences. In a typical text summarization dataset consisting of documents that are 800 tokens in length (average), capturing long-term dependencies is very important, e.g., the last sentence can be grouped with the first sentence of a document to form a summary. LSTMs (Long Short-Term Memory) proved useful for machine translation. However, they often fail to capture long-term dependencies while modeling long sequences. To address these issues, we have adapted Neural Semantic Encoders (NSE) to text summarization, a class of memory-augmented neural networks by improving its functionalities and proposed a novel hierarchical NSE that outperforms similar previous models significantly. The quality of summarization was improved by augmenting linguistic factors, namely lemma, and Part-of-Speech (PoS) tags, to each word in the dataset for improved vocabulary coverage and generalization. The hierarchical NSE model on factored dataset outperformed the state-of-the-art by nearly 4 ROUGE points. We further designed and used the first GPU-based self-critical Reinforcement Learning model.
This submission claims to adapt and improve on neural semantic encoders for the task of abstractive summarization. The submission: - briefly describes the neural semantic encoder (NSE) model (Munkhdalai and Yu, 2017) - claims that the dot-product attention mechanism in NSE is too simplistic for the task of summarization because dot-product attention does not capture word-sentence and sentence-sentence correlations, the latter two being important to summarize a document, and suggests utilizing the additive attention mechanism along with the pointer-generator mechanism instead - Introduces the “hierarchical NSE” model and a self-critical sequence training scheme to optimize for ROUGE - Provides empirical results for these models and for other relevant baseline models on the CNN/Daily Mail dataset At a high-level, this submission is unpolished, imprecise, lacks technical (both experimental and theoretical) rigor in its description and explanation of its methods, and generally does not have a significant contribution, which is why I believe it should be rejected. The primary motivation underlying this submission is strong: modeling long-range dependencies in a document is very critical for abstractive summarization. However, the submission does not make a convincing case for its proposed methods/models, in the context of this motivation and otherwise. The whole premise behind the NSE (according to Munkhdalai and Yu’s paper) is that its variable-sized encoding memory allows it to access the entire input sequence during reading, thus providing for long-range dependencies and helping with tasks like summarization. So the exceptionally low ROUGE scores for plain NSE in Table 2 seemed at odds with the motivation and premise. The authors attribute this very briefly to the dot-product attention mechanism and show empirically that changing it to an additive attention mechanism leads to a significant increase in ROUGE. At an intuitive level, the notion of “highlighting” seems to be congruous to “attending”. The submission (with the title Read, Highlight and Summarize) seems to be suggesting that they are incongruous and that there is novelty in this notion itself, however there is no clear mathematical description or rigorous explanation of this notion. The suggested improvements beyond the standard NSE architecture (augmented with the well-known additive attention and the pointer-generator mechanism) are limited to extending NSE in a “hierarchical” fashion and optimizing for ROUGE, both of which have not been described in sufficient detail or clarity for the task at hand. Munkhdalai and Yu evaluated the NSE approach for document sentiment analysis (among other tasks) and for that task, they used two NSEs: one for obtaining sentence representations and another for document representations using sentence representations. This submission’s “hierarchical NSE” seems to overlap with Munkhdalai and Yu’s approach for document sentiment analysis, except that the document memory seems to be constructed in a different fashion that is not clearly described. In Table 1, Hier-NSE seems to be performing worse than the pointer-generator with coverage loss and only slightly better without coverage loss. Since Hier-NSE contains the pointer-generator mechanism, it seems the Hier-NSE isn’t really adding any non-trivial value. What would be interesting to try out as an exercise is to see if the factoring of lemma and PoS tags described helps significantly improve the pointer-generator with coverage loss. Some typos: - Equation 5 should have m_{r,t}, not m_{t,t} - Page 5 right above equation 22 has a Stack Overflow link that has nothing to do with the submission Suggestions on improving the submission that are here to help, not necessarily part of my decision assessment: - Provide dimensions for every non-scalar parameter in the Proposed Models section: while I see dimensions for some parameters, I do not see dimensions for others - Use accessible notation that makes it easy to distinguish between a non-scalar parameter and a scalar parameter (boldface non-scalars, use capitalized letters for only matrices/tensors, not scalars or vectors, etc.) - Use mathbb{R} for depicting the set of real numbers in LaTeX
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Read, Highlight and Summarize: A Hierarchical Neural Semantic Encoder-based Approach
Traditional sequence-to-sequence (seq2seq) models and other variations of the attention-mechanism such as hierarchical attention have been applied to the text summarization problem. Though there is a hierarchy in the way humans use language by forming paragraphs from sentences and sentences from words, hierarchical models have usually not worked that much better than their traditional seq2seq counterparts. This effect is mainly because either the hierarchical attention mechanisms are too sparse using hard attention or noisy using soft attention. In this paper, we propose a method based on extracting the highlights of a document; a key concept that is conveyed in a few sentences. In a typical text summarization dataset consisting of documents that are 800 tokens in length (average), capturing long-term dependencies is very important, e.g., the last sentence can be grouped with the first sentence of a document to form a summary. LSTMs (Long Short-Term Memory) proved useful for machine translation. However, they often fail to capture long-term dependencies while modeling long sequences. To address these issues, we have adapted Neural Semantic Encoders (NSE) to text summarization, a class of memory-augmented neural networks by improving its functionalities and proposed a novel hierarchical NSE that outperforms similar previous models significantly. The quality of summarization was improved by augmenting linguistic factors, namely lemma, and Part-of-Speech (PoS) tags, to each word in the dataset for improved vocabulary coverage and generalization. The hierarchical NSE model on factored dataset outperformed the state-of-the-art by nearly 4 ROUGE points. We further designed and used the first GPU-based self-critical Reinforcement Learning model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This submission claims to adapt and improve on neural semantic encoders for the task of abstractive summarization. The submission: - briefly describes the neural semantic encoder (NSE) model (Munkhdalai and Yu, 2017) - claims that the dot-product attention mechanism in NSE is too simplistic for the task of summarization because dot-product attention does not capture word-sentence and sentence-sentence correlations, the latter two being important to summarize a document, and suggests utilizing the additive attention mechanism along with the pointer-generator mechanism instead - Introduces the “hierarchical NSE” model and a self-critical sequence training scheme to optimize for ROUGE - Provides empirical results for these models and for other relevant baseline models on the CNN/Daily Mail dataset At a high-level, this submission is unpolished, imprecise, lacks technical (both experimental and theoretical) rigor in its description and explanation of its methods, and generally does not have a significant contribution, which is why I believe it should be rejected. The primary motivation underlying this submission is strong: modeling long-range dependencies in a document is very critical for abstractive summarization. However, the submission does not make a convincing case for its proposed methods/models, in the context of this motivation and otherwise. The whole premise behind the NSE (according to Munkhdalai and Yu’s paper) is that its variable-sized encoding memory allows it to access the entire input sequence during reading, thus providing for long-range dependencies and helping with tasks like summarization. So the exceptionally low ROUGE scores for plain NSE in Table 2 seemed at odds with the motivation and premise. The authors attribute this very briefly to the dot-product attention mechanism and show empirically that changing it to an additive attention mechanism leads to a significant increase in ROUGE. At an intuitive level, the notion of “highlighting” seems to be congruous to “attending”. The submission (with the title Read, Highlight and Summarize) seems to be suggesting that they are incongruous and that there is novelty in this notion itself, however there is no clear mathematical description or rigorous explanation of this notion. The suggested improvements beyond the standard NSE architecture (augmented with the well-known additive attention and the pointer-generator mechanism) are limited to extending NSE in a “hierarchical” fashion and optimizing for ROUGE, both of which have not been described in sufficient detail or clarity for the task at hand. Munkhdalai and Yu evaluated the NSE approach for document sentiment analysis (among other tasks) and for that task, they used two NSEs: one for obtaining sentence representations and another for document representations using sentence representations. This submission’s “hierarchical NSE” seems to overlap with Munkhdalai and Yu’s approach for document sentiment analysis, except that the document memory seems to be constructed in a different fashion that is not clearly described. In Table 1, Hier-NSE seems to be performing worse than the pointer-generator with coverage loss and only slightly better without coverage loss. Since Hier-NSE contains the pointer-generator mechanism, it seems the Hier-NSE isn’t really adding any non-trivial value. What would be interesting to try out as an exercise is to see if the factoring of lemma and PoS tags described helps significantly improve the pointer-generator with coverage loss. Some typos: - Equation 5 should have m_{r,t}, not m_{t,t} - Page 5 right above equation 22 has a Stack Overflow link that has nothing to do with the submission Suggestions on improving the submission that are here to help, not necessarily part of my decision assessment: - Provide dimensions for every non-scalar parameter in the Proposed Models section: while I see dimensions for some parameters, I do not see dimensions for others - Use accessible notation that makes it easy to distinguish between a non-scalar parameter and a scalar parameter (boldface non-scalars, use capitalized letters for only matrices/tensors, not scalars or vectors, etc.) - Use mathbb{R} for depicting the set of real numbers in LaTeX
assistant
This submission claims to adapt and improve on neural semantic encoders for the task of abstractive summarization. The submission: - briefly describes the neural semantic encoder (NSE) model (Munkhdalai and Yu, 2017) - claims that the dot-product attention mechanism in NSE is too simplistic for the task of summarization because dot-product attention does not capture word-sentence and sentence-sentence correlations, the latter two being important to summarize a document, and suggests utilizing the additive attention mechanism along with the pointer-generator mechanism instead - Introduces the “hierarchical NSE” model and a self-critical sequence training scheme to optimize for ROUGE - Provides empirical results for these models and for other relevant baseline models on the CNN/Daily Mail dataset At a high-level, this submission is unpolished, imprecise, lacks technical (both experimental and theoretical) rigor in its description and explanation of its methods, and generally does not have a significant contribution, which is why I believe it should be rejected. The primary motivation underlying this submission is strong: modeling long-range dependencies in a document is very critical for abstractive summarization. However, the submission does not make a convincing case for its proposed methods/models, in the context of this motivation and otherwise. The whole premise behind the NSE (according to Munkhdalai and Yu’s paper) is that its variable-sized encoding memory allows it to access the entire input sequence during reading, thus providing for long-range dependencies and helping with tasks like summarization. So the exceptionally low ROUGE scores for plain NSE in Table 2 seemed at odds with the motivation and premise. The authors attribute this very briefly to the dot-product attention mechanism and show empirically that changing it to an additive attention mechanism leads to a significant increase in ROUGE. At an intuitive level, the notion of “highlighting” seems to be congruous to “attending”. The submission (with the title Read, Highlight and Summarize) seems to be suggesting that they are incongruous and that there is novelty in this notion itself, however there is no clear mathematical description or rigorous explanation of this notion. The suggested improvements beyond the standard NSE architecture (augmented with the well-known additive attention and the pointer-generator mechanism) are limited to extending NSE in a “hierarchical” fashion and optimizing for ROUGE, both of which have not been described in sufficient detail or clarity for the task at hand. Munkhdalai and Yu evaluated the NSE approach for document sentiment analysis (among other tasks) and for that task, they used two NSEs: one for obtaining sentence representations and another for document representations using sentence representations. This submission’s “hierarchical NSE” seems to overlap with Munkhdalai and Yu’s approach for document sentiment analysis, except that the document memory seems to be constructed in a different fashion that is not clearly described. In Table 1, Hier-NSE seems to be performing worse than the pointer-generator with coverage loss and only slightly better without coverage loss. Since Hier-NSE contains the pointer-generator mechanism, it seems the Hier-NSE isn’t really adding any non-trivial value. What would be interesting to try out as an exercise is to see if the factoring of lemma and PoS tags described helps significantly improve the pointer-generator with coverage loss. Some typos: - Equation 5 should have m_{r,t}, not m_{t,t} - Page 5 right above equation 22 has a Stack Overflow link that has nothing to do with the submission Suggestions on improving the submission that are here to help, not necessarily part of my decision assessment: - Provide dimensions for every non-scalar parameter in the Proposed Models section: while I see dimensions for some parameters, I do not see dimensions for others - Use accessible notation that makes it easy to distinguish between a non-scalar parameter and a scalar parameter (boldface non-scalars, use capitalized letters for only matrices/tensors, not scalars or vectors, etc.) - Use mathbb{R} for depicting the set of real numbers in LaTeX