How optimization influences the generalization ability of a DNN is still an active area of research. This work aims to unveil and study a factor of influence: the speed at which each layer trains. In our preliminary work, we develop a visualization technique and an optimization algorithm to monitor and control the layer rotation rate, a tentative measure of layer-level training speed, and show that it has a remarkably consistent and substantial impact on generalization. Our experiments further suggest that weight decay's and adaptive gradients methods' impact on both generalization performance and speed of convergence are solely due to layer rotation rate changes compared to vanilla SGD, offering a novel interpretation of these widely used techniques, and providing supplementary evidence that layer-level training speed indeed impacts generalization. Besides these fundamental findings, we also expect that on a practical level, the tools we introduce will reduce the meta-parameter tuning required to get the best generalization out of a deep network.
Paper summary: The authors propose layer rotation speed as a measure of layer-wise training speed and introduce the Layca optimizer as a means of enforcing uniform layer rotation speed throughout the network. They show empirically that layer rotation speed is linked to the generalization performance of deep neural networks and that weight decay induces uniform layer rotation speeds.
Detailed comments:
Overall, I felt that the paper introduced some interesting ideas but I was not left convinced that layer rotation speed is the correct measure of layer training speed. I hope that the authors can clarify this based on my questions and comments below.
1) In the introduction you refer to input and feedback signals to a layer, I assume this refers to the forward and backward pass. As I understand it, this intuition and the findings of Figure 1 do not immediately relate to the notion of layer rotation speed during training. Could you clarify what you mean by "the input and feedback signals that a layer receives could also influence the generalization ability induced by the layer's training", to me this statement seems obvious as input+feedback signals contains training entirely.
2) In Figure 1 you show that when training one chosen layer and keeping the others fixed, if the chosen layer is deeper into the network the test accuracy is worse. I wonder to what extent this might be remedied by initialization. For example, one might expect that when sampling random square matrices there are some very small eigenvalues which "kill" information in the forward pass. If we train a layer deep in the network it may have access to less information from the data than one earlier on. Whereas training an earlier layer could allow this layer to shift mass into the parts of the eigenspace which are well represented (so-to-speak) in the future layers. Have you thought about this at all? One simple way to evaluate this would be to initialize the weights to be random orthogonal matrices, ensuring that the eigenvalues are equal. With that said, I thought that this was an interesting experiment with a fairly surprising outcome!
3) In related work you discuss the vanishing and exploding gradients problems in terms of layer-level training speed. I think that another relevant research direction may be dynamical isometry [1] which solves this problem by restricting all singular values of the Jacobian matrix to be close to 1. These ideas may also be relevant when discussing Layca and layer-rotation.
4) I found section 3.1. a little unconvincing. It is not obvious to me that layer rotation speed is necessarily a good measure of training speed. In fact, there are many updates which have large cosine distance (as you define it) but do not change the network function (for example, permuting the weight matrices in fully-connected networks). Why is the rotation defined through a vectorization of the weight matrix as opposed to e.g. the polar decomposition? Is this a computational issue? Similarly, in section 3.2 you liken Layca to optimization on a manifold but I am not convinced that this makes sense for matrices which inherently have some structure (e.g, perhaps the Stiefel manifold would be more meaningful).
6) Figure 2 shows that uniform rotation leads to improved test accuracy. But could it be the case that controlling the effective learning rate is sufficient (and layer rotation is one way to achieve this)? For example, we might take the sign of the update and use this to ensure that each weight matrix has the same effective learning rate (something like [2]). Do you expect this would have a similar effect? If not, what is unique about layer rotation that provides good test accuracy?
7) You claim that SGD and adaptive methods with weight decay works without taking extra care to control the layer-rotation rate, as weight decay provides a similar effect. Firstly, you use weight decay and L2 regularization interchangeable, could you be explicit about exactly which you mean (see e.g. [3]). Assuming you mean weight decay (and not L2 regularization), then this could also be due to the effective learning rate ([4,5,6]) which may have some interaction with layer rotation rate (i.e. Figure 4). In summary, I would have liked to see an explanation for why weight decay leads to uniform rotation speeds.
8) If I understand correctly, Figure 5 shows 5 tasks and reportedly 5 optimization schemes - each on a different task? It seems more reasonable to compare these on the same task.
Overall I felt that the paper had some interesting contributions and a fairly comprehensive empirical study. However, I do not feel that the paper gives adequate attention to the notion of effective learning rate induced by weight decay and I was not totally convinced that the way layer rotation speed is defined is the correct way.
Minor comments:
- A lot of white space and a large caption for Figure 1.
- Section 4 opens with "monitor and control", but I think the latter is really presented in section 4 and not section 3.
- I think a diagram of the projection step of the Layca algorithm would be informative (for 2D weight vector).
- Why does `5` appear in equation 1? Is this an arbitrary choice?
- Some of the lighter colors in e.g. Figure 2(b) made some lines hard to read when printed. I do not believe that this affected the image significantly.
Clarity: The paper is well written and is easy to understand. Some of the figures in the experiments are a little cluttered and the lighter colors can be hard to see (e.g. Fig 2(b)), but this is minor.
Significance: The paper presents an interesting view point but I am not convinced that it offers as strong an explanation for these phenomena as other approaches. I believe with some more clarification the results could become more significant. My review score hinges mostly on the interaction between layer rotation speed and the effective layer-wise learning rate.
Originality: To my knowledge, the ideas are presented in the paper are original. In particular, this is a novel way to characterize layer-wise training speed.
References:
[1] Pennington et al. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice" https://arxiv.org/abs/1711.04735
[2] Bernstein et al. "signSGD: Compressed Optimisation for Non-Convex Problems" https://arxiv.org/abs/1802.04434
[3] Loshchilov et al. "Fixing Weight Decay Regularization in Adam", https://arxiv.org/pdf/1711.05101.pdf
[4] Laarhoven, "L2 Regularization versus Batch and Weight Normalization" https://arxiv.org/abs/1706.05350
[5] Hoffer et al. "Norm matters: efficient and accurate normalization schemes in deep networks" https://arxiv.org/abs/1803.01814
[6] Anonymous, "Three Mechanisms of Weight Decay Regularization" https://openreview.net/forum?id=B1lz-3Rct7   (Another ICLR 2019 submission)