Architectural Complexity Measures of Recurrent Neural Networks
In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs). Our main contribution is twofold: first, we present a rigorous graph-theoretic framework describing the connecting architectures of RNNs in general. Second, we propose three architecture complexity measures of RNNs: (a) the recurrent depth, which captures the RNN's over-time nonlinear complexity, (b) the feedforward depth, which captures the local input-output nonlinearity (similar to the "depth" in feedforward neural networks (FNNs)), and (c) the recurrent skip coefficient which captures how rapidly the information propagates over time. We rigorously prove each measure's existence and computability. Our experimental results show that RNNs might benefit from larger recurrent depth and feedforward depth. We further demonstrate that increasing recurrent skip coefficient offers performance boosts on long term dependency problems.
In this work the authors define three different complexity measures for RNNs: recurrent depth, feedforward depth and the recurrent skip coefficient. They give definitions for these three measures within a precise graph-theoretic framework. They then present experiments on various sequential tasks for RNN models with different complexity according to these measures.
A significant portion of the paper is spent defining in a very rigorous way three complexity measures which are fairly obvious, and in my opinion do not need the heavy graph-theoretic machinery the authors make use of.    For example, the authors give intuitive and clear definitions of recurrent depth and skip coefficient (respectively, as the average maximum number of non-linear transformations per time step, and the length of the shortest path from time i to time (i+n)). These are well illustrated with Figure 1 and the formal definitions the authors give in terms of the graph-theoretic framework do not add anything fundamental. Formalizing everything in a rigorous framework could be useful if the authors were proving new and useful theorems, but the theorems they provide are limited to proving the computability of the the measures they define, which seems more of a technical detail.  It is not clear what conclusions to draw from the experiments. The investigation of the different architectures in Figure 2a and 2b seems like an interesting direction, but the experiments are not very systematic and there are some methodological issues.  Questions/comments: -Why do the authors evaluate the architectures in Figure 2a on PennTree and Text8, the architectures in Figure 2b on Text8 and MNIST? Wouldn't it make more sense to evaluate the same set of architectures on all the datasets and see if certain architectures consistently work better on certain types of problems? -are the models whose results are in Table 1 (right) the same size as those reported in Table 1 (left) for Text8? It is not clear if the difference in performance is due to the differences in architecture or model size.  - It is not clear if the improvements with recurrent depth reported in Table 1 (right) are statistically significant, error bars over several initializations would help. - In Section 4.3, the results for MNIST should be included in the main paper, not the appendix. Also, they should be presented in a table like Table 1 (right).  - In Section 4.4, the results on the adding and copying problem should be presented in a table, not the text.  - In Table 2, top left: it is surprising that the results for permuted MNIST are *better* than the unpermuted version, since presumably this increases the timescale of the dependencies. Do you have an explanation? - Sentences like line 294-296: "We varied...found that this kind of improvement persists" should refer to specific results.  It isn't clear what insights the experiments give. The authors say in the conclusion "we find empirical evidence that increasing recurrent depth might yield performance improvements", yet increasing recurrent depth does not help for sequential MNIST and it is not tried for the copy or addition task.  They also say "increasing feedforward depth might not help on long term dependency tasks", which is unsurprising since it should not have a big influence on vanishing or exploding gradients. Their third point is that "increasing recurrent skip coefficient can largely improve performance on long-term dependency task", which was already known (for example, the Clockwork RNN).