In this work, we develop the theory for analyzing the generalization performance of recurrent neural networks. We first present a new generalization bound for recurrent neural networks based on matrix 1-norm and Fisher-Rao norm. The definition of Fisher-Rao norm relies on a structural lemma about the gradient of RNNs. This new generalization bound assumes that the covariance matrix of the input data is positive definite, which might limit its use in practice. To address this issue, we propose to add random noise to the input data and prove a generalization bound for training with random noise, which is an extension of the former one. Compared with existing results, our generalization bounds have no explicit dependency on the size of networks. We also discover that Fisher-Rao norm for RNNs can be interpreted as a measure of gradient, and incorporating this gradient measure not only can tighten the bound, but allows us to build a relationship between generalization and trainability. Based on the bound, we analyze the effect of covariance of features on generalization of RNNs theoretically and discuss how weight decay and gradient clipping in the training can help improve generalization.
This is an interesting and very well-written paper. I read the paper carefully but I don’t have sufficient expertise to determine whether all the proof steps are correct.
This paper builds on recent works giving generalization bounds on RNNs.
The primary contribution is that they are able to bound the generalization error for RNNs without a dependence on the network size parameters d and m.
Their proof is roughly as follows:
1. They decompose the RNN + ReLU activation into the sum of a linear network and difference terms. This step is key because it lets you treat each term independently when estimating the Rademacher complexity.
2. The linear network term can be bounded directly with their Fisher-Rao norm.
3. They make a second decomposition: the difference term can be written as a sum of simpler terms
4. For the simpler terms, their Rademacher complexity can be bounded independently using matrix-1 norm. Using the matrix-1 norm instead of the spectral norm means their bounds won’t depend on the network size parameters.
5. Then they combine these bounds to give the Rademacher complexity bounds for RNNs.
6. Lastly, they combine the Rademacher complexity bound with Kuznetsov et al 2015’s multiclass margin bound to give the generalization bound.
The downside to their generalization bound is that it requires the covariance matrix of the input data must be positive definitive, and it explodes when the smallest eigenvalue is close to zero.
Their second contribution attempts to address these downsides. They prove another generalization bound for RNNs when training with random noise, which has the effect of increasing the term containing the smallest eigenvalue of the input covariance matrix.
They remark on several empirical phenomena that are consistent with their results:
- Correlation of features in the input data makes it harder for RNNs to generalize
- Weight decay could help by decreasing the relevant gradient terms in their bounds
- Gradient clipping could help when the smallest eigenvalue of the input covariance is very small
Their third contribution is a single experiment. This contribution is fairly weak and the practical value of their theoretical work would be much more convincing if they were to put more effort into this section.
- They use IMDB data set (50k movie reviews + binary sentiment classification task)
- They add Gaussian noise to the input data with four different values.
- They plot the generalization error, which is the difference between the test error without noise and the training error with noise
Specific comments to improve their experiment section:
- I’m confused by the following sentence: “generalization errors … for different combinations of L and \sigma_epsilon are shown in Figure 1.” However, in Figure 1 I only see different values of sigma. I don’t see anything about using various values for L. This should be clarified.
- Figure 1 draws linear interpolation between data points - there’s no evidence for those interpolations. It should be reported as a scatter plot, preferably with error bars.
- If space limitations prevent reporting the experiment results more rigorously, I would prefer to see the experiment results reported in an appendix, with a brief comment on their significance in the main paper.