Language Modeling with Graph Temporal Convolutional Networks
Recently, there have been some attempts to use non-recurrent neural models for language modeling. However, a noticeable performance gap still remains. We propose a non-recurrent neural language model, dubbed graph temporal convolutional network (GTCN), that relies on graph neural network blocks and convolution operations. While the standard recurrent neural network language models encode sentences sequentially without modeling higher-level structural information, our model regards sentences as graphs and processes input words within a message propagation framework, aiming to learn better syntactic information by inferring skip-word connections. Specifically, the graph network blocks operate in parallel and learn the underlying graph structures in sentences without any additional annotation pertaining to structure knowledge. Experiments demonstrate that the model without recurrence can achieve comparable perplexity results in language modeling tasks and successfully learn syntactic information.
 The paper applies graph convolutional networks to Penn Treebank language modeling and provides analysis on the attention weight patterns it uses. Clarity: the paper is very clearly written! The introduction states that existing CNN language models are "not easily interpretable in that they do not explicitly learn the structures of sentences". Why is this? The model in this paper computes attention values which is interpreted by the authors as corresponding to the structure of the sentence but there are equivalent means to trace back feature computation in other network topologies as well. My biggest criticism is that the evaluation is done on a very small language modeling benchmark which is clearly out of date. Penn Treebank is the CIFAR10 of language modeling and any claims on this dataset about language modeling are highly doubtful. Models today have tens and hundreds of millions of parameters and training them on 1M words is simply a regularization exercise that does not enable a meaningful comparison of architectures. The claims in the paper could be significantly strengthened by reporting results on at least a mid-size dataset such as WikiText-103, or better even, the One Billion Word benchmark.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Language Modeling with Graph Temporal Convolutional Networks
Recently, there have been some attempts to use non-recurrent neural models for language modeling. However, a noticeable performance gap still remains. We propose a non-recurrent neural language model, dubbed graph temporal convolutional network (GTCN), that relies on graph neural network blocks and convolution operations. While the standard recurrent neural network language models encode sentences sequentially without modeling higher-level structural information, our model regards sentences as graphs and processes input words within a message propagation framework, aiming to learn better syntactic information by inferring skip-word connections. Specifically, the graph network blocks operate in parallel and learn the underlying graph structures in sentences without any additional annotation pertaining to structure knowledge. Experiments demonstrate that the model without recurrence can achieve comparable perplexity results in language modeling tasks and successfully learn syntactic information.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper applies graph convolutional networks to Penn Treebank language modeling and provides analysis on the attention weight patterns it uses. Clarity: the paper is very clearly written! The introduction states that existing CNN language models are "not easily interpretable in that they do not explicitly learn the structures of sentences". Why is this? The model in this paper computes attention values which is interpreted by the authors as corresponding to the structure of the sentence but there are equivalent means to trace back feature computation in other network topologies as well. My biggest criticism is that the evaluation is done on a very small language modeling benchmark which is clearly out of date. Penn Treebank is the CIFAR10 of language modeling and any claims on this dataset about language modeling are highly doubtful. Models today have tens and hundreds of millions of parameters and training them on 1M words is simply a regularization exercise that does not enable a meaningful comparison of architectures. The claims in the paper could be significantly strengthened by reporting results on at least a mid-size dataset such as WikiText-103, or better even, the One Billion Word benchmark.
assistant
 The paper applies graph convolutional networks to Penn Treebank language modeling and provides analysis on the attention weight patterns it uses. Clarity: the paper is very clearly written! The introduction states that existing CNN language models are "not easily interpretable in that they do not explicitly learn the structures of sentences". Why is this? The model in this paper computes attention values which is interpreted by the authors as corresponding to the structure of the sentence but there are equivalent means to trace back feature computation in other network topologies as well. My biggest criticism is that the evaluation is done on a very small language modeling benchmark which is clearly out of date. Penn Treebank is the CIFAR10 of language modeling and any claims on this dataset about language modeling are highly doubtful. Models today have tens and hundreds of millions of parameters and training them on 1M words is simply a regularization exercise that does not enable a meaningful comparison of architectures. The claims in the paper could be significantly strengthened by reporting results on at least a mid-size dataset such as WikiText-103, or better even, the One Billion Word benchmark.