Training deep neural networks (DNNs) has achieved great success in recent years. Modern DNN trainings utilize various types of training techniques that are developed in different aspects, e.g., activation functions for neurons, batch normalization for hidden layers, skip connections for network architecture and stochastic algorithms for optimization. Despite the effectiveness of these techniques, it is still mysterious how they help accelerate DNN trainings in practice. In this paper, we propose an optimization principle that is parameterized by γ > 0 for stochastic algorithms in nonconvex and over-parameterized optimization. The principle guarantees the convergence of stochastic algorithms to a global minimum with a monotonically diminishing parameter distance to the minimizer and leads to a O(1/γK) sub-linear convergence rate, where K is the number of iterations.
Through extensive experiments, we show that DNN trainings consistently obey the γ-optimization principle and its theoretical implications. In particular, we observe that the trainings that apply the training techniques achieve accelerated convergence and obey the principle with a large γ, which is consistent with the O(1/γK) convergence rate result under the optimization principle. We think the γ-optimization principle captures and quantifies the impacts of various DNN training techniques and can be of independent interest from a theoretical perspective.
This paper proposes a "gamma principle" for stochastic updates in deep neural network optimization. First, the authors propose if Eq. (2) is satisfied then convergence is guaranteed (Thm. 1). Second, they use experimental results of Alexnet and Resnet-18 on cifar10/100 to show that Eq. (2) is satisfied by SGD, SGD with momentum, and Adam, with different activations and tricks like batch normalization, skip connection.
Pros:
1. This paper is well written and the presentation is clear.
2. The experiments are extensive.
Cons:
1. Before Eq. (2), it is assumed that over-parameterized NNs are used as \theta. But there is no quantization how many parameters are enough? Are the Alexnet/Resnet-18 in experiments enough over-parameterized and how can we tell that? Some quantitative conditions should be provided to show what kind of models this principle hold.
2. The connection between Eq. (2) and Thm. 1 is too obvious and the gamma is just to characterize the progress of each update. Of course, large progress corresponds to fast convergence. Eq. (2) is a strong assumption rather than a theoretical contribution.
3. Experiments are used to show that Eq. (2) is a "principle". However, the experiments are problematic as follows.
First, "we set \theta^* to be the network parameters produced in the last training iteration", then how do we make sure \theta in the last training iteration is \theta^*, even if the loss is close to (but not exactly) zero? For this point, I suggest using a teacher-student setting, where the optimal \theta^* is already known.
Second, using \theta in the last training iteration makes the experiments show the following simple fact, that methods/tricks/activations with faster convergence to certain parameter will have larger every update progress to that parameter, which is, of course, true and does not reveal an optimization principle of deep learning.
Third, it is difficult to claim that this is a principle for general deep learning by using experiments on two datasets.
Overall, I found the theory not inspiring and experiments not convincing.
========Update=========
Thanks for the rebuttal.
I have read it and using teacher-student setting is an improvement to resolve my question with respect to \theta^*.
However I would maintain my rating since
1) the theoretical contribution is actually marginal;
2) the argument that this "gamma principle" holds for over-parameterized NNs is vague in the sense (and the author did not resolve my concern of this) that for what kinds of over-parameterized NNs this would work and for what kinds of NNs it does not hold. In particular, mentioning other theory work of over-parameterized NNs is not enough, because usually in these work, the numbers of parameters in NNs are poly(n), like O(n^4), O(n^6), where n is number of training data. There is an obvious gap between poly(n) and the number of parameters in experiments. From this perspective, the experiments cannot verify the claim that this "principle" holds for over-parameterized NNs that mentioned by authors in the rebuttal.
3) experiments on two datasets are not enough to claim this is a general "principle".
Considering this claim is for general over-parameterized NN optimization, I think it lacks of specifying types of NNs for which this claim would hold (of course it cannot hold for any NNs, but it possibly can hold for some NNs, and what are these NNs?), and experiments are not enough to show the generality of this claim.