STRUCTURED NEURAL SUMMARIZATION
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text. In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.
Note: I changed my original score from 4 to 7 based on the new experiments that answer many of the questions I had about the relative performance of each part of the model. The review below is the original one I wrote before the paper changes.
# Positive aspects of this submission
- The intuition and motivation behind the proposed model are well explained.
- The empirical results on the MethodNaming and MethodDoc tasks are very promising.
# Criticism
- The novelty of the proposed model is limited since it is essentially adding an existing GGNN layer, introduced by Li et al. (2015), on top of an existing LSTM encoder. The most important novelty seems to be the custom graph representation for these sequence inputs to make them compatible with the GGNN, which should then deserve a more in-depth study (i.e. ablation study with different graph representations, etc).
- Since you compare your model performance against Alon et al. on Java-small, it should be fair to report the numbers on Java-med and Java-large as well.
- The "GNN -> LSTM+POINTER" experiment results are reported on the MethodDoc task, but not for MethodNaming. Reporting this number for MethodNaming is essential to show the claimed empirical superiority of the hybrid encoder compared to GNN only.
- I have doubts about the usefulness of the proposed model for natural language summarization, for the following reasons:
    - The comparison of the proposed model for NLSummarization against See et al. is a bit unfair, since it uses additional information through the CoreNLP named entity recognizer and coreference models. With the experiments listed in Table 1, there is no way to know whether the increased performance is due to the hybrid encoder design or due the additional named entity and coreference information. Adding the entity and coreference data in a simpler way (i.e. at the token embedding level with a basic sequence encoder) in the ablation study would very useful to answer that question.
    - In NLSummarization, connecting sentence nodes using a NEXT edge can be analogous to using a hierarchical encoder, as used by Nallapati et al. ("Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond", 2016). Ignoring the other edges of the GNN graph, what are the theoretical and empirical advantages of your method compared to this sentence-level hierarchical encoder?
    - Adding the coverage decoder introduced by See et al. to your model would have been very useful to prove that the current performance gap is indeed due to the simplistic decoder and not something else.
- How essential is the weighted averaging for graph-level document representation (Gilmer et al. 2017) compared to uniform averaging?
- A few minor comments about writing:
    - In Table 1, please put the highest numbers in bold to improve readability
    - On page 7, the word "summaries" is missing in "the model produces natural-looking with no noticeable negative impact"
    - On page 9, "cove content" should be "core content"