Training and Analyzing Deep Recurrent Neural Networks
Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this paper we study the effect of a hierarchy of recurrent neural networks on processing time series. Here, each layer is a recurrent network which receives the hidden state of the previous layer as input. This architecture allows us to perform hierarchical processing on difficult temporal tasks, and more naturally capture the structure of time series. We show that they reach state-of-the-art performance for recurrent networks in character-level language modeling when trained with simple stochastic gradient descent. We also offer an analysis of the different emergent time scales.
I like the point about the arbitrariness of older data 
      benefiting from potentially many more layer of processing compared to 
      newer data. And building a deep RNN seems like a reasonable way to do 
      better on shorter term stuff, in addition to possible other benefits. 
      Another obvious thing to do would be to put extra layers of nonlinearities 
      between each timestep. Have you tried this? How important was the 
      gradient normalizations and why does this "avoid of the problem of 
      bifurcations"? There is a recent paper in ICML 2013 by researchers from U 
      of Montreal that looks at gradient truncation and optimization in RNNs 
      that that may be relevant here. Also, will normalizing the gradient in 
      this way potentially mess up the "averaging" behavior of SGD? In 
      terms of previous results on these kinds of Wikipedia compression tasks, 
      there is also some work by Mikolov et al. that you may want to compare to. 
      Of the various experiments designed to examine the different roles 
      played by each layer in terms of time-scale and perhaps "abstraction", the 
      one I find most persuasive is the text generation one (as shown in Table 
      2). However, as pointed out by the authors themselves in the paragraph on 
      line 241, it may be problematic to interpret the effect of this kind of 
      "brain surgery" on RNNs due to the complex interdependencies that may have 
      developed between the outputs of the various layers. For me, 
      the biggest missing piece of the empirical puzzle in this paper is the 
      question of whether the higher layers are actually *better* at processing 
      more abstract and long-term properties of the text than if the units were 
      moved to the first layer. i.e. are they benefiting from the extra levels 
      of processing that proceed them in a nontrivial way? That they happen to 
      take on these seemingly more abstract and longer term roles after training 
      is good but incomplete evidence that this is the case. I notice 
      that the regular RNN seems to have less parameters in these experiments 
      since 2*767^2 * 5 > 2119^2 (I'm counting both recurrent and inter-layer 
      weight matrices, hence the multiplication by 2), so the comparison might 
      be a bit unfair. A more convincing would be if the deeper RNNs did better 
      than a standard RNN with the same number of parameters, or even better, 
      the same number of *units*. Say a 2 layer DRNN versus such an RNN with the 
      same number of units, where the different in the number of parameters 
      wouldn't favor the RNN so much that the comparison would be rendered 
      unfair. This would be strengthen the paper's claims a lot in my opinion. 
      Also, instead of Figure 2, it would be better to have seen how 
      well various depths of DRNN did on the benchmarks when trained from 
      scratch, possibly with wider layers than their deeper counterparts to make 
      up for the difference the numbers of parameters and/or units. 
      Minor: - You should define DRNN-AO and IO in the text 
      somewhere and not just in Figure 1.