Recently, there have been some attempts to use non-recurrent neural models for language modeling. However, a noticeable performance gap still remains. We propose a nonrecurrent neural language model, dubbed graph temporal convolutional network (GTCN), that relies on graph neural network blocks and convolution operations. While the standard recurrent neural network language models encode sentences sequentially without modeling higher-level structural information, our model regards sentences as graphs and processes input words within a message propagation framework, aiming to learn better syntactic information by inferring skip-word connections. Specifically, the graph network blocks operate in parallel and learn the underlying graph structures in sentences without any additional annotation pertaining to structure knowledge. Experiments demonstrate that the model without recurrence can achieve comparable perplexity results in language modeling tasks and successfully learn syntactic information.
This paper draws inspiration from recent works on graph convolutional networks and proposes GTCN, a convolutional architecture for language modeling. The key intuition is to treat sentences as (potentially densely-connected) graphs over tokens, instead of sequences as in many RNN-based language models. The model then, when predicting a token, summarizes previous tokens using attention mechanism as context. Empirical evaluation on word-level language modeling on Penn Treebank shows competitive performance.
The idea of this work appears reasonable and well-motivated to me. But the connections to previous works, especially those based on self-attention, should be clearly addressed. Further, writing can be improved, and I would encourage a thorough revision since there are typos making the paper a bit hard to follow.
Last but not least, I find several of the claims not very-well supported. Please see details below.
Pros:
- Well-motivated intuition treating language as structured.
Cons:
- Writing can be improved.
- Missing discussion of existing works. 
Details:
- Based on my understanding of Eqs. 6--11, the proposed GTCN seems to be a gated version (also equipped with window-2 convolutions) of the self-attention mechanism. Could the authors comment on how GTCN relates to Vaswani et al. (2017), Salton et al. (2017), among others? Also, empirical comparisons to self-attention based language models might be necessary.
- I was confused by Eqs. 13--14 and the text around it. Doesn't one need some kind of classifier (e.g., an MLP) to predict x_{t+1}? Why are these two equations predicting word embedding?
- The start of Section 4.1. There seems to be a typo here. I'm assuming the two vectors are `$\mathbf{v}$ and $\mathbf{q}$` here, as in Eqs. 6 and 7.
- More clarification on Eq. 9 might be necessary. Is \mathbf{W}^p part of the parameters? I'm guessing \mathbf{W}_{i-j}^p selects a row from the matrix, since there is a dot product outside.
- Can the authors clarify Eq. 5? I'm not sure how to interpret it, and it seems not used anywhere else.
- Eq. 2 is a bit misleading: it might give the impression that f_{t+1} does not depend on f_t (and so forth), which is not the case for LSTM.
- It would be interesting to be how GTCN compare to other models in efficiency, since the paper mentions parallel computation many times.
- Contribution.2: GTCN is not really the state-of-the-art model on LM.
- Comparison to RNNG: RNNG treats each sentence as a separate sequence, in contrast to most cited works in Table 1, where the whole training (eval) set is treated as a single sequence, and truncate the length when applying BPTT. And according to the second paragraph of Section 5.1, this work follows the latter. To the best of my knowledge, such a difference does have an effect on the perplexity metric. In this sense, RNNG is not comparable to the rest in Table 1. It is perhaps fine to still put it in the table, but please clarify it in the text.
Minors:
- Why is the margins above equations seem larger. Can the authors make sure the template is right?
- Around Eq.5: why is \mathbf{X} is capitalized in the eq, but not in the text? Are they the same thing?
- Section 4.3: the dependence of attention weights $a$ is not reflected in the notation.
- Section 5.1: I think what it means here is a `10K` vocabulary, instead of a 10K word tiny corpora.
References
Vaswani et al.. 2017. Attention is All You Need. In Proc. of NIPS.
Salton et al.. 2017. Attentive Language Models.