The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence. To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors. Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings. We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs. Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.
This work proposes an end-to-end graph encoder to sequence decoder model with an attention mechanism in between.
Pros (+) :
+ Overall, the paper provides a good first step towards flexible end-to-end graph-to-seq models.
+ Experiments show promising results for the model to be tested in further domains.
Cons (-) :
- The paper would benefit more motivation and organization.
Further details below (+ for pros / ~ for suggestions / - for cons):
The paper could benefit a little more motivation:
- Mentioning a few tasks in the introduction may not be enough. Explaining why these tasks are important may help. What is the greater problem the authors are trying to solve?
- Same thing in the experiments, not well motivated, why these three? What characteristics are the authors trying to analyze with each of these tasks?
Rephrase the novelty argument:
- The authors argue to present a “novel attention mechanism” but the attention mechanism used is not new (Bahdanau 2014 a & b). The fact that it is applied between a sequence decoder and graph node embeddings makes the paper interesting but maybe not novel.
~ The novelty added by this paper is the “bi-edge-direction“ aggregation technique with the exploration of various pooling techniques. This could be emphasized more.
Previous work:
~ The Related Work section could mention Graph Attention Networks (https://arxiv.org/abs/1710.10903) as an alternative to the node aggregation strategy.
Aggregation variations:
+ The exploration between the three aggregator architectures is well presented and well reported in experiments.
~ The two Graph Embedding methods are also well presented, however, I didn’t see them in experiments. Actually, it isn’t clear at all if these are even used since the decoder is attending over node embeddings, not graph embedding… Could benefit a little more explanation
Experiments:
+ Experiments show some improvement on the proposed tasks compared to a few baselines.
- The change of baselines between table 1 for the first two tasks and table 2 for the third task is not explained and thus confusing.
~ There are multiple references to the advantage of using “bi-directional” node embeddings, but it is not clear from the description of each task where the edge direction comes from. A better explanation of each task could help.
Results:
- Page 9, the “Impact of Attention Mechanism” is discussed but no experimental result is shown to support these claims.
Some editing notes:
(1) Page 1, in the intro, when saying “seq2seq are excellent for NMT, NLG, Speech Reco, and drug discovery”: this last example breaks the logical structure of the sentence because it has nothing to do with NLP.
(2) Page 1, in the intro, when saying that “<...> a network can only be applied to sequential inputs”: replace network by seq2seq models to be exact.
(3) Typo on page 3, in paragraph “Neural Networks on Graphs”, on 8th line “usig” -> “using”
(4) Page 3, in paragraph “Neural Networks on Graphs”, the following sentence: “An extension of GCN can be shown to be mathematically related to one variant of our graph encoder on undirected graphs.” is missing some information, like a reference, or a proof in Appendix, or something else…
(5) Page 9, the last section of the “Impact of Hop Size” paragraph talks about the impact of the attention strategy. This should be moved to the next paragraph which discusses attention.
(6) Some references are duplicates:
|_ Hamilton 2017 a & c
|_ Bahdanau 2014 a & b