Neural language models have achieved great success in many NLP tasks, to a large extent, due to the ability to capture contextual dependencies among terms in a text. While many efforts have been devoted to empirically explain the connection between the network hyperparameters and the ability to represent the contextual dependency, the theoretical analysis is relatively insufficient. Inspired by the recent research on the use of tensor space to explain the neural network architecture, we explore the interpretable mechanism for neural language models. Specifically, we define the concept of separation rank in the language modeling process, in order to theoretically measure the degree of contextual dependencies in a sentence. Then, we show that the lower bound of such a separation rank can reveal the quantitative relation between the network structure (e.g. depth/width) and the modeling ability for the contextual dependency. Especially, increasing the depth of the neural network can be more effective to improve the ability of modeling contextual dependency. Therefore, it is important to design an adaptive network to compute the adaptive depth in a task. Inspired by Adaptive Computation Time (ACT), we design an adaptive recurrent network based on the separation rank to model contextual dependency. Experiments on various NLP tasks have verified the proposed theoretical analysis. We also test our adaptive recurrent neural network in the sentence classification task, and the experiments show that it can achieve better results than the traditional bidirectional LSTM.
This paper first provides a theoretical interpretation of separation rank as a measure of a recurrent network's ability to capture contextual dependencies in natural language text. The analysis is primarily done in the context of tensor space language models (TSLM) that was demonstrated to be a generalisation of n-gram language models in earlier work. The theoretical derivations suggest that increasing a network's depth increases its separation rank exponentially, while increasing its width only increases its separation rank linearly. Based on this finding, the paper proposes a bidirectional adaptive recurrent neural network that adapts the network depth for each task using dynamic halting. Experiments on six NLP tasks demonstrate that increasing network depth helps more for tasks that generally require many long-range dependencies, while increasing network width is generally sufficient for tasks that require mostly short-term dependencies, although I disagree with the paper's dichotomy of tasks with long-term and short-term dependencies (see point 3 below).
Overall, this paper suffers from several serious issues, as listed below. I am therefore recommending a rating of "Reject" ahead of the authors' response.
1. This paper suffers from substantial clarity issues beyond simple grammatical errors. Some examples of the most serious clarity issues are as follows: (i) Section 6.1.1 lists "sentiment analysis" as one of the tasks, but Figure 2 has no entry for "sentiment analysis", yet instead features experimental results for "semantic classifier" which was never mentioned or introduced before; and (ii) Figure 2 shows "1st layer LSTM", "2nd layer LSTM", and "3rd layer LSTM", while (based on my reading) what the paper means are "1-layer LSTM", "2-layer LSTM", and "3-layer LSTM", hence highly confusing for the reader.
2. The proposed explanation about the proposed bidirectional adaptive RNN is really sparse, despite being a central part of the paper. The explanation of the proposed model is only contained in two paragraphs (Sections 5.1 and 5.2), and are not sufficient for the readers to understand the model. A more extensive explanation and intuition about what the model is like, and how it is similar or different to TSLM and standard RNNs, is required to improve this.
3. In the experiments, the paper assumes a false dichotomy of tasks that only require short-range dependencies (NER, POS tagging, and constituency parsing), and tasks that require long-range dependencies (WSD, sentiment analysis, and coreference resolution). This dichotomy is overly simplistic and ultimately false. For instance, constituency parsers often need to identify spans that are very long-distance in nature (for a recent investigation of this, see the work by Fried et al. (2019)), while sentiment analysis often only requires the model to identify a few salient words that are indicative of the sentiment, e.g. "excellent" or "terrible", hence not requiring much contextual dependencies. 
4. Related to point 3, a better evaluation is to examine the cases that require long-range dependencies within each task, rather than assuming which tasks require long-range dependencies and which ones do not. An example of this is reporting e.g. constituency parsing performance for long-range spans and coreference resolution performance for long-distance entity chains.
5. The paper mostly fails to report performance comparison with existing numbers from prior work. For instance, the coreference performance (Fig. 2) are far below the result from Lee et al. (2017) that this paper is based on (Section 6.1.1), while the constituency parsing numbers (also Fig. 2) are also far below the reconciled span parser (Joshi et al., 2018) that this paper is also based on. This discrepancy calls into question the strength of the model implementation used in this paper.
6. Some missing citations, e.g. the use of adaptive computation time and dynamic halting in Universal Transformers (Dehghani et al., 2019).
7. This paper can benefit from careful copy-editing. Some examples of grammatical errors: (i) "Eq. 16" in Section 5.2 should be "Eq. 6", (ii) "to verify our theoretical, respectively" in Section 6 should be "to verify our theoretical [findings/derivations]", (iii) "ourself" on the caption of Table 1 should be "ourselves", (iv) "The model Ling et al. (2015) is used, which using bidirectional LSTMs" should be "The model [of] Ling et al. (2015) is used, which [used] bidirectional LSTMs", etc.
References
Daniel Fried, Nikita Kitaev, and Dan Klein. "Cross-domain generalization of neural constituency parsers". In Proc. of ACL 2019.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. "End-to-end neural coreference resolution". In Proc. of EMNLP 2017.
Vidur Joshi, Matthew E. Peters, and Mark Hopkins. "Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples". In Proc. of ACL 2018.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. "Universal Transformers". In Proc. of ICLR 2019.