Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation as well as the spectral properties of the state-to-state Jacobians. By optimizing these quantities in terms of the initialization hyperparameters, we derive a novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe a beneficial effect on generalization performance using this new initialization.
The aim of this paper is to suggest randomized initializations for the various weights of a recurrent neural network (GRUs and various LSTMs are covered), such that training these networks gets to a successful start, when the model is trained on long sequences. Instead of being heuristic, their approach follows first principles of analyzing signal propagation through time, using ideas from statistical thermodynamics (mean field approximations). Some experiments, on toy datasets, validate their approach.
I am quite intrigued by this paper. It is using interesting theory, shapes it to a practically highly relevant and difficult applied problem, and in the end comes up with a computable criterion of how to choose hyperparameters (means and variances of Gaussians to sample initial weights from). While the results in practice are still not too convincing, I am strongly in favour of giving this approach the benefit of doubt, as it could lead to practically very useful downstream work.
The main direction of improvement for this paper (given that experiments are what they are -- somewhat limited to toy situations right now) is to better explain the methodology to researchers not familiar with mean field methods. Most importantly, it is not explained in the main text how hyperparameters are really chosen in the end. Looking at Appendix E, I find some pretty basic choices, and no other alternatives considered. It is not explained why these choices satisfy the theory, why they'd be the only ones, etc. This creates a disconnect between the very nice (and seemingly useful) theory and its implications (they are not really well spelled out).
Here is what I understood (and I am not specifically an expert on stat mech). The authors assume that the dimension of latent states (N) grows large. They assume that weights are sampled independently, and identical distributed in groups k (different cell types, weight vs bias), and that inputs are correlated with each other in each dimension. Based on these assumption, they follow Gaussians statistics through a number of time steps. In the limit, one gets a deterministic dynamical system, and as t -> infty, this may converge to a fixed point. In a very nice argument (which they could explain better), they state that such rapid convergence is bad news, because then information cannot spread across long time scales, so one has to find hyperparameters for which the system behaves "critically". A second arguments tries to keep gradient sizes (under MF assumptions) of O(1), so neither -> 0, not -> infty, which is again some "critical" range. Under their assumption, these critical conditions can be computed depending on the hyperparameters.
Unfortunately, this is where the paper somewhat stops, it does not give specific methods for finding hyperpars that satisfy the criteria, at least not in the sense of characterising the whole space of such hyperpars (instead, in Appendix E, they just state some few settings that do). As a direction for future work, this would be very important. Another side question is whether for what the authors call "trainability", the only point that matters is whether for the initial weights, signals can spread and gradients are O(1). It is certainly necessary, I see that.
Detailed comments:
- Please fix Table 1, the expressions seem broken. What does "r2" mean in the GRU column?
- At least for me, (1a) to (1c) really was too short. At least in the Appendix, please do explain how this gives GRU and LSTM
- Please explain the untied weight assumption somewhere. s^t is a map of s^(t-1) and W_k, so how can W_k be independent
   of all s^t? What are you really assuming here?
- It took some repeated reading until I understood why the expressions in (2a) to (4b) do not depend on i, j, a, b (except
  whether a = b or a != b). Explain that properly
- The core of the whole approach seems to be first half of page 5. This seems like a very nice argument, but hard to understand. Try making it more crisp. I kind of get the rough idea why fast convergence over t would be bad, but would total divergence over t not also be bad?
- In (12a-c), do you mean "equal" or "approximately equal"?
- In 4.4: "This motivates the general form of the initializations used in the experiments": You have to make this more explicit. Why are your choices the only ones? Could there not be other choices satisfying (12a-c) approx, and be better?
- Value of Sigma_z = 1: This seems odd to me, then your covariances are degenerate (rank 1 instead of 2). Please explain
- Standard LSTM harder than GRU or peephole LSTM: Again, this sounds real interesting, but I did not get it from the explanation
- I did not understand Figure 2. How are Theta_0, Theta_1 chosen?
- As said above, the experiments are interesting, but somewhat artificial. Please do at least comment on real-world applications, and whether (and how) the ideas here would apply
- Discussion: "there is no clear principled way...": Well, but practitioners need something. I'd disagree, at least one could attempt to navigate this space by global optimization techniques...
ADDITIONAL COMMENT:
I tried to append the following as comment, but the (pretty broken) system would not let me, insisting that "reader is not valid" (???). Anyway, here it is. I hope I am allowed to add to my own review.
I've seen the argument in reviews that assumptions made by this paper about independencies between weights and inner states are wrong, and therefore conclusions are not valid.
First, such assumptions are indeed pretty common in such statistical mech analyses of learning methods. Second, you have to distinguish between weights after (random) initialization and after training. Of course, LSTM represents long term dependencies after training, but initialization is a different story.
If I was the AC for this paper, I'd ask somebody with at least some background in statistical mech to provide some additional opinions, as the reviewers (including myself) are not fully qualified.